:W&Bá¾@©.„M uub1bs2mª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0<"S Á‚" avsgh8tgq2x9 uub1bs2mŠ uub1bs2mspatop" _wandb‚{}Bdp_dsfrac_070_outputj dp-spatop-train-dsfrac-070-5xhlwŠ Ãø‰Ó¨Þš1 (H¨¸h"3.12.10*0.28.1j linux-x86_64;.ÿ`ÒÚ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 Âø‰Ó¸¿°¨*-of*@/data/spatop/options_files/sweep_fixed_dp/config_dsfrac_070.json*-l*/scratch/logs/*-n*dp_dsfrac_070_output2-m spanet.trainRdanielprimosch@gmail.comZ/scratch/logs/b dp-spatop-train-dsfrac-070-5xhlwr/opt/conda/bin/pythonº 3g1mueailcc56yyvyyszrlji7hbuaretèŒÛW2  requirements.txtiƒÆP uub1bs2mZ? (H¨¸ (H¨¸h"3.12.10*0.28.1j linux-x86_64uÓ¹?R uub1bs2mZA (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64kÞæŸ* uub1bs2mb trainer/global_step2ÃåºjU uub1bs2mZD (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64Øôà­/  uub1bs2mb*"trainer/global_step2ObÂK  uub1bs2mj: Äø‰Ó˜§€Ö*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] Éh™úÇ ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€´ð7€À‡¤'ª€À½­ÝÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-972543b9-faeb-2f81-bd6a-2715e03d5530Ê13.2º 3g1mueailcc56yyvyyszrlji7hbuaret.Ÿ“  uub1bs2mj  Äø‰ÓРá¶î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 5.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 75.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ S%¶ \  uub1bs2mjÊ Äø‰Ó¸ãô··Trainable params: 84.5 M Non-trainable params: 50 Total params: 84.5 M Total estimated model params size (MB): 337.925 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 ÞWõ³¾ uub1bs2m*¬  hidden_dim‚256  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256  num_embedding_layers‚6  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚0.7  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  batch_size‚2048  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW" ( learning_rate‚0.00011712902303919398  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0  dropout‚0.26251405799488203 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"X˜â_ë uub1bs2mjÙ Äø‰ÓØÈ¥¼È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. 1„>N uub1bs2mj¼ Äø‰Ó¨œ¹¼«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ÷Ð]°ë uub1bs2mjÙ Èø‰ÓøÒ…ÉÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. º?P uub1bs2mj¾ Èø‰Ó°·–É­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. §8k–‚(:“ Óø‰Ó¬”â³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.942886352539055* gpu.0.memoryAllocatedBytes‚ 49772560384 gpu.0.temp‚42.0 gpu.0.powerWatts‚256.529' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.50966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hô"+‚(:¡  Óø‰Ó¿ù§µ% disk./.usageGB‚1257.0566940307617 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.sda2.out‚ 1.1171875 disk.drbd2041.out‚0 disk.drbd1817.out‚0% memory_percent‚45.064353942871094 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.30119096105659 disk.sda.in‚ 0.00390625 network.sent‚42440 network.recv‚41509 disk.sda.out‚ 1.1171875 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 0.6484375 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 0.1259765625 disk.drbd1510.in‚0* proc.memory.percent‚11.370118459065754 disk.drbd1938.in‚0 disk.dm-0.out‚ 0.44140625 proc.cpu.threads‚17 disk.sda2.in‚ 0.00390625 disk.drbd1584.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0" proc.memory.rssMB‚ 2794.3203125 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 0.6484375 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0( proc.memory.availableMB‚ 13500.984375 cpu‚20.214430727934904 disk.drbd2078.out‚0 disk.nvme0n1.out‚ 0.44140625 disk.drbd1584.in‚0iÞNœ‚(:’ âø‰Óݿdz gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.95265197753906* gpu.0.memoryAllocatedBytes‚ 49780948992 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.422' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.14066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Ÿˆj+‚(:¡  âø‰Óó¸í³! disk.nvme0n1.out‚ 7.6552734375 disk.drbd1550.in‚0 disk.dm-0.out‚ 7.6552734375 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.sda.in‚ 0.00390625 disk.sda2.out‚ 1.4296875 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 0.6484375 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 cpu‚21.410421849164646 disk.sda1.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.372979482014973' proc.memory.availableMB‚ 13500.46875 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2795.0234375 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0( disk./.usagePercent‚70.3011920277458 disk.drbd1285.out‚0 disk.sda2.in‚ 0.00390625 disk.drbd1938.out‚0$ memory_percent‚45.06645202636719 network.sent‚42770 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 0.6484375 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 0.1259765625 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.sda.out‚ 1.4296875 disk.drbd1931.in‚0 network.recv‚41839$ disk./.usageGB‚1257.056713104248 disk.drbd1673.in‚0ÝE~ #‚(:™  ñø‰ÓýÖß³ disk.drbd1817.in‚0 cpu‚22.14934851860027 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.00390625* proc.memory.availableMB‚13500.11328125 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 0.1884765625! disk.nvme0n1.out‚ 7.8427734375 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.sda2.in‚ 0.00390625 disk.drbd1965.out‚0 network.recv‚43350 disk.drbd2111.out‚0 disk.drbd1226.out‚0 network.sent‚43478 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.sda.out‚1.84375 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚1.4375 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.dm-0.in‚1.4375 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.sda2.out‚1.84375 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2795.0234375( disk./.usagePercent‚70.3011920277458 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.dm-0.out‚ 7.8427734375 disk.drbd1510.in‚0$ memory_percent‚45.06789843241374* proc.memory.percent‚11.372979482014973$ disk./.usageGB‚1257.056713104248 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0vêÇ ‚(:“ ñø‰ÓáØ¿³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.955093383789055* gpu.0.memoryAllocatedBytes‚ 49783046144 gpu.0.temp‚48.0 gpu.0.powerWatts‚189.28' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.093333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QuS'œ‚(:’ €ù‰ÓÁÛ¢³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.95753479003907* gpu.0.memoryAllocatedBytes‚ 49785143296 gpu.0.temp‚53.0 gpu.0.powerWatts‚268.193' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.39766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§/‚(:¥  €ù‰ÓÇðƳ disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 0.1884765625 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.00390625" proc.memory.rssMB‚ 2795.0390625* proc.memory.percent‚11.373043060302734 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 2.4453125 cpu‚22.6462112611086% disk./.usageGB‚1257.0566940307617 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.00390625 disk.drbd2118.out‚0 proc.cpu.threads‚17 network.sent‚43892 network.recv‚43764 disk.dm-0.in‚ 2.4453125 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 2.25390625 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0" disk.nvme0n1.out‚ 9.18408203125) disk./.usagePercent‚70.30119096105659 disk.drbd1673.out‚0 disk.drbd1356.out‚0& proc.memory.availableMB‚ 13500.0625 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0% memory_percent‚45.068105061848954 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.dm-0.out‚ 9.18408203125 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.sda.out‚ 2.25390625 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0†» &‚(:œ  ù‰Ó¿Ôå³ disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 2.4453125 disk.drbd2111.in‚0 network.recv‚44028 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 cpu‚23.00796303206073 network.sent‚44156 disk.sda.in‚0.015625 disk.sda.out‚2.78125 disk.drbd1931.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.30119096105659 disk.nvme0n1.in‚ 2.4453125 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda2.out‚2.78125" disk.nvme0n1.out‚ 9.32470703125 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 13499.9296875 disk.drbd1285.in‚0 disk.drbd1289.out‚0! proc.memory.rssMB‚ 2795.046875 proc.cpu.threads‚17% disk./.usageGB‚1257.0566940307617 disk.rbd1.out‚ 0.400390625* proc.memory.percent‚11.373074849446613 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.dm-0.out‚ 9.32470703125$ memory_percent‚45.06864547729492 disk.drbd1289.in‚0 disk.drbd1828.out‚0Ò©œ‚(:’ ù‰Ó٪ij gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚55.0 gpu.0.powerWatts‚270.175' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.05833333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q…ì®3b/ lr-AdamW/pg2"trainer/global_step2:JPyZH”3b/ lr-AdamW/pg1"trainer/global_step2:JP±NÔ;-) _wandbruntime‚89  _runtime‚89Wcå ½ ' lr-AdamW/pg1‚2.0497579031858943e-05 ' lr-AdamW/pg2‚2.0497579031858943e-05 trainer/global_step‚49 % _timestamp‚1.7848393266006267e+09 _runtime‚ 89.959054691 _step‚0à¬PˆÄ!¿ % _timestamp‚1.7848393266006267e+09 _runtime‚ 89.959054691 _step‚0 ' lr-AdamW/pg1‚2.0497579031858943e-05 ' lr-AdamW/pg2‚2.0497579031858943e-05 trainer/global_step‚49¹è'çk"bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPy«|g#bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPðú%^$bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP µíe%ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPøéÿßm&bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP5PûXo'bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPò ¹i(be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPLøj3p)bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPšÀG k*bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPcDÔjj+bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPšrJs,bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP “•lq-bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP®Ó=eq.bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP m[>o/bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP1Ÿ2l0bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPJar\f1bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPg¶Úh2bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPÚC‚Èi3be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP@3†$s4bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP2LÙ’q5bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPj›,Ãm6bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPh¿L`7b\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP*H—mf8bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPÖ2eþe9ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPb úc:b_ bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPã˜s?bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP kËjh@bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP;#íŒxAbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP¸*ݹvBbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP(õ°kCbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPá—‹âjDbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPb¶"×eEba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPXÂ5ÐrFbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÛTŒ.nGbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPâ­nHbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP’n ×mIbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP5¤ò(jJbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPÈhiceKba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP§ËôDqLbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPeÈÚfMbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP©¸“õfNbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPè¹³mWbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPœá”dXb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP¨Ë›¢kYbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPDæ5¹jZbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP?½¾n[bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPé¨Ûq\bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP·¨1‡]]bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPâúej^bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPEÎÿÔa_b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPæц _`b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP–@Knabj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPôZPobbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP¥@¦’gcbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP¯r‰_db[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPóÆüUgebc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP^äcfb_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPõkýj„be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPо\#`…b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP²ím†bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPÈçñi‡bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPÚbÂyˆbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPþ‹¤t‰bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP™”“´lŠbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP·SƒÞk‹bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP~pÙomŒbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP+HL‹ibd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP˜»Ô¿hŽbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPi—º%hbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP¯ôlKubp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPK°‹„n‘bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPÅÚ¿(e’b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPU o“bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP°ÎËra”b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP9T.g•bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPseW«q–bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP%øÏo—bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP Àîj˜be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPà×q™bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPˆàfÌgšbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP(Õœ¿b›b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPr=4Ædœb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14083187055616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚55 trainer/global_step‚49 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423706112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚422764 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 51978733056 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345282560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14032553605120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9764 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14084558511616 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 51892042044 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344610034 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1723858944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7848393266015217e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345282560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4184254143488 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370038002 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11922 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚617 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49239031808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14032579778560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 50547432010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚422758 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371456000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680017920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚585 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709566976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680017920 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚452871 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚137166 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4184677849600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371456000 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9770 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚875629 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚422764 _runtime‚ 89.959169923 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 50633450496 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885399 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2003 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14078773880908‚åfö©'Ÿ£O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11922 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 50633450496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14026856410896 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14084558511616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11997 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4119544540160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9764 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14078773880908 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14077403842906 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 64765218304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14026881838864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49239031808 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚462635 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1379186176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚290111 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 64709603328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2003 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚452871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679330790 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49239031808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1723858944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚822 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14083187055616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚55 trainer/global_step‚49 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423706112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚422764 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 51978733056 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345282560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14032553605120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9764 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14084558511616 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 51892042044 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344610034 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1723858944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7848393266015217e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345282560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4184254143488 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370038002 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11922 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚617 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49239031808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14032579778560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 50547432010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚422758 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371456000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680017920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚585 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709566976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680017920 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚452871 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚137166 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4184677849600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371456000 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9770 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚875629 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚422764 _runtime‚ 89.959169923 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 50633450496 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885399 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2003 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP%R»~m«bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPc[þv_¬bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPW¯*^­bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPh 6{k®bf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPI e¯b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP­5êÆf°ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP¤5j±be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPÃRÈ$r²bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP-’aBi³bd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP]“EÇ]´bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP”Ú=ƒgµbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP{fXm¶bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP\^Ïn·bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP%\že¸b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPŒc&]`¹b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPc8ìdVºbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPŒÔÒœh»bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPrCêœp¼bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPM¦,ñe½b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP˜—‰d¾b_ 9oÂbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPÓúUbiÃbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP¹y‘aÄb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPîø{kÅbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPÚÚÑQdÆb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP#€³{eØb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP3Œ,ºlÙbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP¬{nnábi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP*¹_olâbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPêE0©uãbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP6:çoäbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP;¹ã"påbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPÓjgæbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPïNpiçbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JP8æ9Õmèbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPu»kqébl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÊОcêb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPø˜lêmëbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPƒ,·)`ìb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPçïíš^íbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPôÈ7úuîbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPjïjïbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP-ø¨neðb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP¹.hñbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPÐd[fbòb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPRfnóbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP(ïVmôbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPyè¢hõbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPãzéföba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPéE¤^÷bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPƒ,doøbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPKcÏtwùbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP ¦{nrúbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPêH‘Ömûbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPÊë=lübg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP ƒ’ÿpýbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP³¤²‚fþba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP]´@9lÿbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP%:çr€bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPÀt¢wbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPªÓf‚ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP€aƒb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP}1f„ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP ̸Ko…bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP³FÒõi†bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPê2›g‡bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP^î_Íkˆbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPÐøfg‰bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP·š©Ö;Šbn KDeviceStatsMonitor.on_train_batch_end/inactive_split†9“n8_bytes.small_pool.freed"trainer/global_step2:JPFùf‹ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP•“ŽÞhŒbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP*@7hbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP㪆ècŽb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPVÉ×?db_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚295757 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚155953 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2003 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14358596456436 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚461953 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚296098 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 318088704 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11999 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421601792 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚431125 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚617 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14305683612640 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 51655958528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1723858944 _step‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14311467089920 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 51655958528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚55 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345289728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679330798 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1723858944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709568000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53510656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49239031808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49239031808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14305658184672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚893078 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 53001248256 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 66008375808 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 52912843796 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚431131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚585 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4264511567360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14311493263360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4198503191552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2003 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14311493263360 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚461953 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14357226418322 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14364494511616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14363123048448 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 65954865152 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902862 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚341 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49239031808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2003 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚617 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14305683612640 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 51655958528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1723858944 _step‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14311467089920 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 51655958528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚55 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345289728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679330798 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1723858944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709568000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53510656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49239031808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49239031808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14305658184672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚893078 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 53001248256 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 66008375808 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 52912843796 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚431131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚585 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4264511567360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14311493263360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4198503191552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2003 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14311493263360 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚461953 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14357226418322 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14364494511616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14363123048448 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 65954865152 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902862 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚341 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49239031808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2003 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚295757 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚155953 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2003 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14358596456436 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚461953 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚296098 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 318088704 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11999 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421601792 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚431125 F loss/SRqqt2/assignment_loss"trainer/global_step2:JP#›G}A¤b< loss/FBt2/assignment_loss"trainer/global_step2:JPõNT±@¥b; loss/FBt1/detection_loss"trainer/global_step2:JP‡ì@¦b; loss/FRt2/detection_loss"trainer/global_step2:JP’˜oB§b= loss/SRqqt1/detection_loss"trainer/global_step2:JP`Ý–ÍB¨b= loss/SRqqt2/detection_loss"trainer/global_step2:JP“ÈyŠ@©b; loss/FBt2/detection_loss"trainer/global_step2:JP´7_ï7ªb2 loss/total_loss"trainer/global_step2:JP£Ýù¹A«b< loss/FRt1/assignment_loss"trainer/global_step2:JP…³ûÁC¬b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPÊãßA­b< loss/FBt1/assignment_loss"trainer/global_step2:JP; ¬Š.®) _wandbruntime‚91  _runtime‚91yÒœö¯ð 0loss/FBt1/assignment_loss‚1.8698363313034398 _runtime‚ 91.469119837 _step‚3 /loss/FRt2/assignment_loss‚5.359197204304683 .loss/FRt1/detection_loss‚0.355476018265165 epoch‚0 1loss/SRqqt2/assignment_loss‚4.220181375544068 /loss/FBt2/assignment_loss‚1.933108401298523 .loss/FBt1/detection_loss‚9.749183313677838 trainer/global_step‚49 $ _timestamp‚1.784839328111847e+09 /loss/FRt2/detection_loss‚1.7089956797218038 0loss/SRqqt1/detection_loss‚5.605939700144033 1loss/SRqqt2/detection_loss‚1.4539744797722989 .loss/FBt2/detection_loss‚27.20168522387743 %loss/total_loss‚68.19815601463688 /loss/FRt1/assignment_loss‚4.485531828681967 2loss/SRqqt1/assignment_loss‚4.2550464580456415&–ÛJú°ô /loss/FRt1/assignment_loss‚4.485531828681967 2loss/SRqqt1/assignment_loss‚4.2550464580456415 0loss/FBt1/assignment_loss‚1.8698363313034398 _runtime‚ 91.469119837 _step‚3 /loss/FRt2/assignment_loss‚5.359197204304683 .loss/FRt1/detection_loss‚0.355476018265165 epoch‚0 1loss/SRqqt2/assignment_loss‚4.220181375544068 /loss/FBt2/assignment_loss‚1.933108401298523 .loss/FBt1/detection_loss‚9.749183313677838 trainer/global_step‚49 $ _timestamp‚1.784839328111847e+09 /loss/FRt2/detection_loss‚1.7089956797218038 0loss/SRqqt1/detection_loss‚5.605939700144033 1loss/SRqqt2/detection_loss‚1.4539744797722989 .loss/FBt2/detection_loss‚27.20168522387743 %loss/total_loss‚68.19815601463688V°[±‚(:’ ­ù‰Ó°§«³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚59.0 gpu.0.powerWatts‚272.822' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.94066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ûB+²‚(:   ­ù‰Ó¯ŠÉ³ disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2795.1015625 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 network.recv‚48455 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.sda.out‚ 3.4453125 cpu‚23.49684844754101 disk.sda2.out‚ 3.4453125 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 network.sent‚60456$ disk./.usageGB‚1257.055835723877 disk.drbd1931.in‚0 disk.dm-0.out‚ 9.98388671875 disk.drbd2118.in‚0) proc.memory.availableMB‚ 13498.9609375 disk.rbd1.out‚ 0.642578125 disk.sda2.in‚0.015625 disk.dm-0.in‚ 2.4453125 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.sda.in‚0.015625* proc.memory.percent‚11.373297373453777) disk./.usagePercent‚70.30114296004241 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1673.out‚0$ memory_percent‚45.07258733113606 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 2.4453125 disk.drbd1226.out‚0 disk.drbd2118.out‚0" disk.nvme0n1.out‚ 9.98388671875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0Áëf³‚(:’ ¼ù‰ÓùÒé³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚60.0 gpu.0.powerWatts‚203.905' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.96833333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý,,´‚(:¡  ¼ù‰Ó¹Þ€´ disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 network.sent‚60856 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 network.recv‚48785 disk.rbd1.out‚ 0.642578125 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.sda.out‚ 3.8984375% disk./.usageGB‚1257.0568084716797 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.015625 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.30119736119181 disk.drbd1817.out‚0 disk.dm-0.out‚ 10.2958984375 disk.drbd1931.in‚0 cpu‚23.669515715870634" disk.nvme0n1.out‚ 10.2958984375 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 2.4453125 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.drbd1356.in‚0$ memory_percent‚45.07265090942383 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 13498.9453125! proc.memory.rssMB‚ 2795.109375 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 2.4453125 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 3.8984375* proc.memory.percent‚11.373329162597656 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0ÚWê µ‚(:’ Ëù‰Ó•Ú»³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚62.0 gpu.0.powerWatts‚298.156' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.38533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03 -0¶‚(:¥  Ëù‰Ó•êس disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2795.11328125* proc.memory.percent‚11.373345057169598 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0) disk./.usagePercent‚70.30124877561147 disk.sda2.in‚0.015625 disk.drbd1938.in‚0 disk.drbd1965.out‚0 network.recv‚50432 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 2.4453125 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 4.2421875 disk.drbd2041.out‚0" disk.nvme0n1.out‚ 10.9248046875 disk.drbd1510.in‚0 disk.sda2.out‚ 4.2421875 disk.dm-0.in‚ 2.4453125 disk.drbd1965.in‚0 cpu‚23.810199209552692 network.sent‚69188 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 0.7685546875 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0$ memory_percent‚45.07308006286621 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.dm-0.out‚ 10.9248046875 disk.drbd2111.out‚0% disk./.usageGB‚1257.0577278137207 disk.drbd1552.out‚0 disk.sda1.out‚0* proc.memory.availableMB‚13498.83984375 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1931.in‚0ÁX·‚(:’ Úù‰Ó¿Œß³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚63.0 gpu.0.powerWatts‚296.515' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.83833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09~ Ö0¸‚(:¥  Úù‰Óï‚´ disk.drbd2118.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 13498.2578125) disk./.usagePercent‚70.30124877561147 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 cpu‚23.930446307768968 disk.drbd1817.in‚0 disk.rbd1.out‚ 0.7685546875 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 4.58203125 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.sda.in‚0.015625 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 2.4453125 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0! disk.nvme0n1.out‚ 11.220703125 disk.drbd1550.out‚0 disk.dm-0.in‚ 2.4453125 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 network.sent‚119351 disk.sda.out‚ 4.58203125 disk.drbd1938.in‚0 disk.drbd1828.in‚0 network.recv‚53113 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0% disk./.usageGB‚1257.0577278137207 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1226.in‚0 disk.sda2.in‚0.015625 disk.dm-0.out‚ 11.220703125# proc.memory.rssMB‚ 2795.12109375* proc.memory.percent‚11.373376846313477 disk.drbd1285.in‚0 disk.drbd2041.in‚0$ memory_percent‚45.07544835408528 disk.drbd1289.in‚0 disk.drbd2078.in‚0bz|ʹ‚(:’ éù‰ÓºÚ«³ gpu.0.gpu‚50.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚64.0 gpu.0.powerWatts‚295ÕÂîåÓ.655' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.55166666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úFz(º‚(:  éù‰Ó¢þг disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0* proc.memory.availableMB‚13498.24609375 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0# memory_percent‚45.0754960378011* proc.memory.percent‚11.373392740885418 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.dm-0.out‚ 11.388671875 disk.drbd1584.in‚0 disk.drbd1226.out‚0% disk./.usageGB‚1257.0575141906738 disk.drbd1510.in‚0 network.sent‚125548 disk.rbd1.out‚ 0.8310546875 disk.nvme0n1.in‚ 2.4453125 disk.sda.in‚0.015625 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 cpu‚24.03271369962974 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.sda.out‚ 4.9765625 proc.memory.rssMB‚2795.125 disk.sda2.in‚0.015625 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 network.recv‚55460) disk./.usagePercent‚70.30123682869238 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0! disk.nvme0n1.out‚ 11.388671875 disk.drbd1673.in‚0 disk.sda2.out‚ 4.9765625 disk.dm-0.in‚ 2.4453125 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17üC0»+ _wandbruntime‚176  _runtime‚176§çH«Â¼¼ & lr-AdamW/pg1‚4.141347600314358e-05 & lr-AdamW/pg2‚4.141347600314358e-05 trainer/global_step‚99 % _timestamp‚1.7848394133985627e+09 _runtime‚ 176.755874805 _step‚4 ;  ƽÀ & lr-AdamW/pg1‚4.141347600314358e-05 & lr-AdamW/pg2‚4.141347600314358e-05 trainer/global_step‚99 % _timestamp‚1.7848394133985627e+09 _runtime‚ 176.755874805 _step‚4« T0¾+ _wandbruntime‚176  _runtime‚176µÒ¾L²'¿¬O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28081422307840 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28080050851840 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27978254018560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8049072552960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚841108 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚841114 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚269070 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚5 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288314368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 % _timestamp‚1.7848394133997645e+09 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚571855 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679330798 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49239031808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2003 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11999 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚302789 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49239031808 trainer/global_step‚99 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚269066 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28081422307840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421608960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53517824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9764 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27966945099696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1723858944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 103168289280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 176.757384404 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1749931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚55 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370038002 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371456000 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9764 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27978227845120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345282560 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 127324280832 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 102995924852 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚918587 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8049440644096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28069966452516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28080050851840 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9770 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚908823 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚908823 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8176764924928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1118855168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28068596414514 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709568000 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 103168289280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚572216 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759701 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚841108 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101823006720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680018944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11924 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667564503 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49239031808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1723858944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27978227845120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚601 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 327389696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101651314818 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚822 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚822 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27978254018560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2003 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚918587 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚303146 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11999 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 127270763008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28080050851840 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27978254018560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8049072552960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚841108 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚841114 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚269070 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚5 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288314368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 % _timestamp‚1.7848394133997645e+09 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚571855 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679330798 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49239031808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2003 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11999 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚302789 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49239031808 trainer/global_step‚99 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚269066 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28081422307840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421608960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53517824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9764 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27966945099696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1723858944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 103168289280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 176.757384404 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1749931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚55 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370038002 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371456000 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9764 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27978227845120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345282560 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 127324280832 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 102995924852 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚918587 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8049440644096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28069966452516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28080050851840 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9770 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚908823 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚908823 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8176764924928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1118855168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28068596414514 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709568000 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 103168289280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚572216 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759701 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚841108 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101823006720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680018944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11924 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667564503 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49239031808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1723858944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27978227845120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚601 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 327389696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101651314818 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚822 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚822 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27978254018560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2003 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚918587 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚303146 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11999 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 127270763008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777164 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚917905 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8256607358464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1723858944 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28361358307840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚271708 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345289728 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚927683 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777164 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28361358307840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 128576154112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45122048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚822 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28257141329920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49239031808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288314368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11924 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 104190804480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102672116458 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚349 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680018944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11999 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371463168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1387573760 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 128531032064 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28349789028044 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28245772301440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚633 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102845514752 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚849481 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚917905 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2003 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚577468 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767380 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚849481 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 104190804480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28257141329920 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 104016726604 _step‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679330798 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2003W¸7»}ùM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767380 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚849481 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 104190804480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28257141329920 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 104016726604 _step‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679330798 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2003 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767380 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413213184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344610146 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚55 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345289728 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102845514752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚849475 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 327389696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28245746873472 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777164 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚917905 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8256607358464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1723858944 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28361358307840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚271708 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345289728 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚927683 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777164 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28361358307840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 128576154112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45122048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚822 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28257141329920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49239031808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288314368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11924 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 104190804480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102672116458 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚349 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680018944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11999 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371463168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1387573760 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 128531032064 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28349789028044 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28245772301440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚633 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102845514752 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚849481 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚917905 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2003 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5774681õï0Ä+ _wandbruntime‚178  _runtime‚178õ‰X@ûÅõ .loss/FBt1/detection_loss‚6.831194566476876 epoch‚0 trainer/global_step‚99 $ _timestamp‚1.784839414911717e+09 _runtime‚ 178.268904022 _step‚7 /loss/FRt1/assignment_loss‚4.524789238764235 2loss/SRqqt1/assignment_loss‚3.8171490020439274 2loss/SRqqt2/assignment_loss‚3.8069263605985735 1loss/FBt1/assignment_loss‚0.29824015554348543 0loss/FBt2/assignment_loss‚0.5240487153508834 0loss/FRt1/detection_loss‚0.36860283353624845 /loss/FRt2/detection_loss‚1.5203969644470103 1loss/SRqqt2/detection_loss‚1.2757954507213431 .loss/FBt2/detection_loss‚37.96836143732071 /loss/FRt2/assignment_loss‚5.318203433888071 $loss/total_loss‚71.6314101711052 0loss/SRqqt1/detection_loss‚5.377702012413838¥é8mÿÆù /loss/FRt2/assignment_loss‚5.318203433888071 $loss/total_loss‚71.6314101711052 0loss/SRqqt1/detection_loss‚5.377702012413838 .loss/FBt1/detection_loss‚6.831194566476876 epoch‚0 trainer/global_step‚99 $ _timestamp‚1.784839414911717e+09 _runtime‚ 178.268904022 _step‚7 /loss/FRt1/assignment_loss‚4.524789238764235 2loss/SRqqt1/assignment_loss‚3.8171490020439274 2loss/SRqqt2/assignment_loss‚3.8069263605985735 1loss/FBt1/assignment_loss‚0.29824015554348543 0loss/FBt2/assignment_loss‚0.5240487153508834 0loss/FRt1/detection_loss‚0.36860283353624845 /loss/FRt2/detection_loss‚1.5203969644470103 1loss/SRqqt2/detection_loss‚1.2757954507213431 .loss/FBt2/detection_loss‚37.968361437320719J Ç‚(:’ øù‰Óçâà³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚66.0 gpu.0.powerWatts‚269.782' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.92733333333332 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙÛ)«+È‚(:   øù‰Ó‹Ò‚´ disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.dm-0.out‚ 11.9052734375 proc.cpu.threads‚17) disk./.usagePercent‚70.30124024209783 disk.nvme0n1.in‚ 2.4453125 disk.drbd1931.out‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.373392740885418 disk.sda1.out‚0 disk.sda2.out‚ 5.23828125 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1965.in‚0 cpu‚24.116924765000306 network.recv‚55790 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 0.8310546875 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0# memory_percent‚45.0772762298584 proc.memory.rssMB‚2795.125 disk.sda.in‚0.015625 network.sent‚125878 disk.sda1.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0* proc.memory.availableMB‚13497.80859375 disk.drbd1550.out‚0 disk.drbd2118.out‚0# disk./.usageGB‚1257.05757522583 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.015625 disk.drbd2041.in‚0" disk.nvme0n1.out‚ 11.9052734375 disk.drbd2118.in‚0 disk.sda.out‚ 5.23828125 disk.drbd1673.in‚0 disk.dm-0.in‚ 2.4453125RËæžÉ‚(:“ ‡ú‰Ó¢žÛ³ gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚67.0 gpu.0.powerWatts‚301.826' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.60866666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f©ñ)Ê‚(:ž  ‡ú‰ÓÛ…û³ disk.drbd2078.out‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1817.in‚0$ memory_percent‚45.07737159729004# proc.memory.rssMB‚ 2795.12890625 disk.dm-0.out‚ 12.2177734375 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.015625 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.dm-0.in‚4.8125* proc.memory.availableMB‚13497.78515625 disk.drbd1828.in‚0 disk.nvme0n1.in‚4.8125 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1938.out‚0 network.sent‚129143) disk./.usagePercent‚70.30124045543567% disk./.usageGB‚1257.0575790405273 disk.sda2.out‚ 5.8828125 disk.drbd1552.in‚0 cpu‚24.194557924820117" disk.nvme0n1.out‚ 12.2177734375* proc.memory.percent‚11.373408635457356 disk.drbd2111.out‚0 disk.sda.out‚ 5.8828125 network.recv‚57075 disk.rbd1.out‚ 0.95703125 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0¸éË‚(:”  –ú‰ÓÍî׳ disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.sda.out‚ 6.2109375 disk.drbd2078.out‚0 proc.cpu.threads‚17 network.sent‚129473 disk.rbd1.in‚0 disk.nvme0n1.in‚4.8125 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.373408635457356 disk.rbd1.out‚ 0.95703125 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 cpu‚24.261651537910033 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 6.2109375 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 network.recv‚57405) disk./.usagePercent‚70.30124152212488 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚4.8125' proc.memory.availableMB‚ 13497.78125# proc.memory.rssMB‚ 2795.12890625 disk.sda.in‚0.015625 disk.drbd1673.in‚0 disk.dm-0.out‚ 12.484375 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0% disk./.usageGB‚1257.0575981140137 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.out‚ 12.484375 disk.drbd1550.out‚0 disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1289.in‚0% memory_percent‚45.077387491861984 disk.drbd1226.in‚0 disk.drbd1584.out‚0E'>íÌ‚(:’ –ú‰Ó—€¶³ gpu.0.gpu‚38.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚66.0 gpu.0.powerWatts‚287.567' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.85566666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç~“?Í‚(:’ ¥ú‰ÓæØ̳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚68.0 gpu.0.powerWatts‚285.352' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.11733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÚ²+΂(:   ¥ú‰Ó×Åî³ disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.out‚ 12.69921875 disk.dm-0.in‚ 9.6171875 network.recv‚57777 disk.drbd1552.in‚0 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2795.12890625$ disk./.usageGB‚1257.065444946289 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0% memory_percent‚45.077403386433915 network.sent‚129845 disk.sda.out‚ 6.69921875 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 cpu‚24.32183945016406 disk.rbd1.out‚ 1.01953125 disk.nvme0n1.in‚ 9.6171875 disk.drbd1552.out‚0 disk.sda2.out‚ 6.69921875 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.373408635457356( disk./.usagePercent‚70.3016803580635 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 12.69921875 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.015625 disk.drbd2041.out‚0 disk.drbd1828.out‚0* proc.memory.availableMB‚13497.77734375 disk.sda.in‚0.015625 disk.drbd1356.in‚04H•)Ï‚(:ž  ´ú‰Ó¡Äñ³% memory_percent‚45.077451070149735 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0* proc.memory.percent‚11.373408635457356 disk.sda.out‚7.1875 network.sent‚133130 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2795.12890625 network.recv‚59050 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 9.6171875 disk.drbd2111.out‚0) disk./.usagePercent‚70.30193017667513 disk.sda1.in‚0 disk.sda2.in‚0.015625 disk.sda2.out‚7.1875 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1550.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.015625 disk.drbd2041.out‚0 disk.rbd1.out‚ 1.01953125( proc.memory.availableMB‚ 13497.765625 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0$ disk./.usageGB‚1257.069911956787# disk.nvme0n1.out‚14.64990234375 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.dm-0.out‚14.64990234375 disk.drbd1965.out‚0 disk.drbd1828.out‚0 cpu‚24.373630415113166 disk.nvme0n1.in‚ 9.6171875 disk.drbd1817.in‚0 disk.drbd1289.in‚0ÑѸЂ(:’ ´ú‰ÓÑÒÔ³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚67.0 gpu.0.powerWatts‚295.095' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.36500000000001 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¾žÑ‚(:“ Ãú‰ÓǕʳ gpu.0.gpu‚75.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚65.0 gpu.0.powerWatts‚111.928' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.309333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸoB5Ò‚(:ª  Ãú‰Ó™ê³ disk.drbd1285.in‚0# disk.nvme0n1.out‚15.07568359375 disk.dm-0.out‚15.07568359375 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd2118.in‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.373408635457356 disk.sda.in‚0.015625 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 1.1455078125 disk.drbd1552.in‚0 disk.drbd1931.out‚0 network.recv‚59899 disk.sda2.in‚0.015625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0% memory_percent‚45.077546437581375) proc.memory.availableMB‚ 13497.7421875 proc.cpu.threads‚17 disk.drbd1817.out‚0) disk./.usagePercent‚70.30192910998593% disk./.usageGB‚1257.0698928833008 disk.sda.out‚ 7.55859375 disk.drbd1938.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 7.55859375 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 9.6171875 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 cpu‚24.419185595774177 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 9.6171875 disk.drbd1965.in‚0 disk.drbd1673.in‚0 network.sent‚133805 disk.drbd2078.in‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2795.12890625GîÏ30Ó+ _wandbruntime‚263  _runtime‚263òÙCÁÔ» & lr-AdamW/pg1‚6.232937297442822e-05 & lr-AdamW/pg2‚6.232937297442822e-05 trainer/global_step‚149 % _timestamp‚1.7848395003728993e+09 _runtime‚ 263.7301314 _step‚8À ûöÅÕ¿ & lr-AdamW/pg1‚6.232937297442822e-05 & lr-AdamW/pg2‚6.232937297442822e-05 trainer/global_step‚149 % _timestamp‚1.7848395003728993e+09 _runtime‚ 263.7301314 _step‚8÷Î#$0Ö+ _wandbruntime‚263  _runtime‚263OÖo"׸O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42078286104064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚340 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2003 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 189966976512 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9770 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2624233 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1259464 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1364775 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371456000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371456000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11999 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1364775 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 154357845504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680018944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚638 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11978600565760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667564503 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42059788986122 $ _timestamp‚1.784839500373939e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11999 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41923928258560 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 153012562944 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41923902085120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9764 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚853983 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚400966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚ 263.731492984 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634003 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1259458 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42076914648064 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152755197626 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345282560 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚400970 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1387573760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 190010008576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49239031808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚822 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12168978665472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1723858944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚9 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1374539 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1259464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1259458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚822 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288314368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚606 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 411123200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41923928258560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1723858944 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9764 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42078286104064 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 154357845504 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42076914648064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41923902085120 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1374539 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚853643 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344610034 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.al$ÒºâOl.current‚2003 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚453013 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚43032064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288314368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚606 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 411123200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41923928258560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1723858944 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9764 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42078286104064 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 154357845504 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42076914648064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41923902085120 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1374539 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚853643 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344610034 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2003 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚453013 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚43032064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41923902085120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9764 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚853983 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚400966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚ 263.731492984 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634003 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1259458 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42076914648064 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152755197626 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345282560 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚400970 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1387573760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 190010008576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49239031808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚822 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12168978665472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1723858944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚9 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1374539 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1259464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1259458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚822 ¨h0Ù+ _wandbruntime‚265  _runtime‚265í;-%Ê&ÚÄM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1383635 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709568000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49239031808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚822 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42339611561538 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2003 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚859538 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288314368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 154035070976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11924 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚149 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1267825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1118855168 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 191215557120 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42202841743360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2641682 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1267825 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚403604 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12248406683136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49316352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344610146 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 265.214676201 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 155380360704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚10 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12057191126016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1373857 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42358222104064 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚403608 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚455930 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚455591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417407488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42340981599652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚822 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9778 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚859195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641682 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1267831 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42202815569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651466 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚606 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 191264873472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1723858944 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Âë.RÎ&ÛÈM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1373857 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42358222104064 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚403608 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚455930 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚455591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417407488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42340981599652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚822 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9778 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚859195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641682 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1267831 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42202815569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651466 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚606 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 191264873472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1723858944 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1383635 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709568000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49239031808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚822 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42339611561538 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2003 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚859538 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288314368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 154035070976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11924 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚149 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1267825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1118855168 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 191215557120 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42202841743360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2641682 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1267825 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚403604 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12248406683136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49316352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344610146 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 265.214676201 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 155380360704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚10 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12057191126016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ±!D0Ü+ _wandbruntime‚265  _runtime‚265ܪþÝø 0loss/FBt1/assignment_loss‚0.2677023814542702 0loss/FRt1/detection_loss‚0.44789710315919573 /loss/FRt2/detection_loss‚1.5422406735256688 %loss/total_loss‚43.11019521753687 % _timestamp‚1.7848395018928583e+09 0loss/FBt2/assignment_loss‚0.3349009345333975 0loss/SRqqt1/detection_loss‚4.795676399092103 epoch‚0 trainer/global_step‚149 /loss/FRt1/assignment_loss‚4.579470712793542 2loss/SRqqt1/assignment_loss‚2.5487379125887983 1loss/SRqqt2/assignment_loss‚2.476217429776218 1loss/SRqqt2/detection_loss‚1.0676097738884103 .loss/FBt2/detection_loss‚16.28411154351186 _runtime‚ 265.250135776 /loss/FBt1/detection_loss‚3.4321780255435743 _step‚11 /loss/FRt2/assignment_loss‚5.333452327669829âq[ÁÞü /loss/FRt1/assignment_loss‚4.579470712793542 2loss/SRqqt1/assignment_loss‚2.5487379125887983 1loss/SRqqt2/assignment_loss‚2.476217429776218 1loss/SRqqt2/detection_loss‚1.0676097738884103 .loss/FBt2/detection_loss‚16.28411154351186 _runtime‚ 265.250135776 /loss/FBt1/detection_loss‚3.4321780255435743 _step‚11 /loss/FRt2/assignment_loss‚5.333452327669829 0loss/FBt1/assignment_loss‚0.2677023814542702 0~pîîAloss/FRt1/detection_loss‚0.44789710315919573 /loss/FRt2/detection_loss‚1.5422406735256688 %loss/total_loss‚43.11019521753687 % _timestamp‚1.7848395018928583e+09 0loss/FBt2/assignment_loss‚0.3349009345333975 0loss/SRqqt1/detection_loss‚4.795676399092103 epoch‚0 trainer/global_step‚149  PFá#ß‚(:˜  Òú‰ÓÒóÕ³$ memory_percent‚45.08002599080403* proc.memory.percent‚11.373408635457356 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.dm-0.out‚ 16.8671875 disk.drbd1285.out‚0 network.sent‚134069 disk.drbd1817.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.30193017667513 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2795.12890625 cpu‚24.4619052871862 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.sda2.in‚0.015625 disk.drbd1828.in‚0 disk.nvme0n1.in‚9.765625 disk.drbd2041.out‚0 disk.sda.out‚8.421875 disk.drbd2078.in‚0 disk.rbd1.out‚ 1.1455078125 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚8.421875 disk.drbd1938.in‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 13497.1328125 network.recv‚60163 disk.dm-0.in‚9.765625 disk.drbd1965.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.out‚ 16.8671875 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0$ disk./.usageGB‚1257.069911956787T”ñÔà‚(:’ Òú‰ÓÁ寳 gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚69.0 gpu.0.powerWatts‚279.338' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.11266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ž›×á‚(:’ áú‰Óæس gpu.0.gpu‚65.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚68.0 gpu.0.powerWatts‚308.01' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.66999999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ËÆ|'â‚(:œ  áú‰Ó°™þ³ proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.dm-0.out‚ 17.0390625 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.30193017667513 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚9.765625 disk.nvme0n1.out‚ 17.0390625 disk.drbd1552.out‚0 disk.sda.out‚ 8.9453125 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 1.2080078125 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 network.recv‚62834 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1931.in‚0 cpu‚24.499740271961517 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 network.sent‚179134 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.dm-0.in‚9.765625 disk.drbd2078.out‚0 disk.rbd1.in‚0$ memory_percent‚45.08199691772461 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.sda2.out‚ 8.9453125# proc.memory.rssMB‚ 2795.12890625* proc.memory.percent‚11.373408635457356 disk.drbd1285.out‚0) proc.memory.availableMB‚ 13496.6484375$ disk./.usageGB‚1257.069911956787 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.015625"ÙôÜã‚(:’ ðú‰Ó‚Û¥³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚68.0 gpu.0.powerWatts‚292.015' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.33833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øw1ä‚(:¦  ðú‰Ó”“dz disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.015625 network.sent‚237016 disk.sda1.in‚0 disk.dm-0.out‚17.41943359375 disk.drbd1510.in‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2795.12890625 disk.dm-0.in‚9.765625 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 9.15234375 disk.drbd1673.in‚0* proc.memory.percent‚11.373408635457356 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 13496.390625% disk./.usageGB‚1257.0698585510254 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0) disk./.usagePercent‚70.30192718994537 disk.drbd1289.in‚0# disk.nvme0n1.out‚17.41943359375 cpu‚24.53278806744397 disk.drbd2111.out‚0 disk.nvme0n1.in‚9.765625 disk.drbd1965.out‚0 network.recv‚65796 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.drbd1673.out‚0% memory_percent‚45.083045959472656 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sda2.out‚ 9.15234375 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 1.2080078125 disk.drbd2118.out‚07Nfå‚(:’ ÿú‰ÓÙÿͳ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚66.0 gpu.0.powerWatts‚294.296' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.09866666666667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ,æ‚(:¡  ÿú‰Ó€Çñ³ disk.drbd1965.out‚0) disk./.usagePercent‚70.30192612325617 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.sda.out‚ 9.8203125 disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0$ disk./.usageGB‚1257.069839477539 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.dm-0.out‚18.02490234375 proc.cpu.threads‚17 disk.rbd1.out‚ 1.419921875 disk.drbd1510.out‚0 disk.drbd1552.out‚0' proc.memory.availableMB‚ 13496.40625 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1828.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚9.765625 disk.drbd2078.in‚0# disk.nvme0n1.out‚18.02490234375 disk.sda.in‚0.015625 disk.sda2.out‚ 9.8203125 network.sent‚237322 disk.drbd2118.out‚0 disk.drbd1931.out‚0 cpu‚24.565032611355498 disk.rbd1.in‚0 disk.nvme0n1.in‚9.765625 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2795.12890625 network.recv‚66102 disk.drbd2078.out‚0 disk.drbd1673.in‚0$ memory_percent‚45.08298238118489* proc.memory.percent‚11.373408635457356??Gwç‚(:’ Žû‰Óê¾Ó³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚65.0 gpu.0.powerWatts‚114.898' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.29933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jI•&2è‚(:§  Žû‰Ó‚¯ó³ disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 1.783203125# disk.nvme0n1.out‚18.51904296875 disk.dm-0.in‚9.765625 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.373424530029297 disk.drbd1550.in‚0 disk.sda1.in‚0 network.recv‚67300) disk./.usagePercent‚70.30192612325617 disk.drbd1510.out‚0 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2795.1328125 disk.drbd1226.out‚0 disk.sda.in‚0.015625 disk.drbd2078.in‚0 disk.drbd1285.out‚0$ disk./.usageGB‚1257.069839477539* proc.memory.availableMB‚13496.32421875 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 10.23046875 disk.dm-0.out‚18.51904296875 disk.drbd1356.in‚0 disk.nvme0n1.in‚9.765625 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0$ memory_percent‚45.08331616719563 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 cpu‚24.593973522070964 disk.drbd2118.in‚0 disk.sda.out‚ 10.23046875 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 network.sent‚240579 disk.drbd2111.in‚0¬Õíé‚(:’ û‰ÓþÇß³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚67.0 gpu.0.powerWatts‚273.086' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.02866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„AŽÔ/ê‚(:¤  û‰ÓÝ×ù³ disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1289.out‚0 cpu‚24.620616798363905 proc.cpu.threads‚17 network.sent‚240885 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 10.6015625 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 1.966796875 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.dm-0.out‚18.68310546875 disk.nvme0n1.in‚9.765625 disk.drbd1673.in‚0 disk.drbd1931.in‚0% disk./.usageGB‚1257.0698623657227 disk.rbd1.in‚0# disk.nvme0n1.out‚18.68310546875 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 13496.3203125 network.recv‚67606 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd2041.out‚0 disk.sda.in‚0.015625 disk.drbd1289.in‚0* proc.memory.percent‚11.373424530029297( disk./.usagePercent‚70.3019274032832 disk.drbd2111.out‚0 disk.drbd1584.in‚0$ memory_percent‚45.08333206176758 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0" proc.memory.rssMB‚ 2795.1328125 disk.dm-0.in‚9.765625 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 10.6015625 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0¯Ò­-0ë+ _wandbruntime‚351  _runtime‚351b;íÃì½ trainer/global_step‚199 $ _timestamp‚1.784839587689601e+09 _runtime‚ 351.046912351 _step‚12 & lr-AdamW/pg1‚8.324526994571287e-05 & lr-AdamW/pg2‚8.324526994571287e-05:¶[ŠÇíÁ _step‚12 & lr-AdamW/pg1‚8.324526994571287e-05 & lr-AdamW/pg2‚8.324526994571287e-05 trainer/global_step‚199 $ _timestamp‚1.784839587689601e+09 _runtime‚ 351.046912351 òA:0î+ _wandbruntime‚351  _runtime‚351v@ÙÂ'ï¼O YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204202119168 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚602217 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16161335788032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1387573760 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56075149900288 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1118855168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56050981557730 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49239031808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55869576325120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11999 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1677814 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415317504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1723858944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371456000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1677808 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 252791891968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667564503 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55847122477296 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚606 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56052351595732 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55869602498560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55869576325120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚638 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498535 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508305 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11999 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2003 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 351.051956235 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1677808 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 trainer/global_step‚199 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2003 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49239031808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15908128578560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1723858944 % _timestamp‚1.7848395876907938e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1135087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚55 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288314368 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 205547401728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345282560 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚532866 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11924 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚532870 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚822 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1820727 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 252839118336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1830491 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49239031808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2003 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚822 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329400832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344610034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚13 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56073778444288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345282560 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56073778444288 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 205203690468 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1820727 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709568000 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56075149900288 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709568000 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1830491 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1723858944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 205547401728 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 204202119168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16160920470528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47226368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667564503 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55847122477296 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚606 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56052351595732 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55869602498560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55869576325120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚638 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498535 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508305 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11999 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2003 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 351.051956235 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1677808 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 trainer/global_step‚199 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2003 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49239031808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15908128578560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1723858944 % _timestamp‚1.7848395876907938e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1135087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚55 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288314368 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 205547401728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345282560 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚532866 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11924 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚532870 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚822 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1820727 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 252839118336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1830491 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49239031808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2003 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚822 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329400832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344610034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚13 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56073778444288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345282560 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56073778444288 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 205203690468 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1820727 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709568000 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56075149900288 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709568000 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1830491 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1723858944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 205547401728 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 204202119168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16160920470528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47226368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2003 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1118855168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56050981557730 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49239031808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55869576325120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDevil†‡ªîceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11999 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1677814 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415317504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 Éí0ñ+ _wandbruntime‚352  _runtime‚352õT™"Ë&òÅM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚55 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1829809 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16240766904832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49316352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1723858944 _step‚14 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1686175 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1387573760 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56148489809920 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345289728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56148489809920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16241184312320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1723858944 $ _timestamp‚1.784839589173068e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1140667 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 206569916928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1829809 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚822 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1839587 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56355085900288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205224627200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 254097082368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49239031808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚822 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3515984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚606 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15987087229952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1686181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680018944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚535508 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1686181 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚604813 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49239031808 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 205224627200 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1839587 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49239031808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56148515983360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345289728 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3515984 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 254047766016 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56330804133146 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56353714437120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56332174171260 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 206224492220 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2003 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206569916928 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚535504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56125924251072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9778 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11924 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288314368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417407488 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1140317 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1686175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709568000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344610146 trainer/global_step‚199 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2003 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0£ÃyóÏ&óÉM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2003 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206569916928 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚535504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56125924251072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9778 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11924 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288314368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417407488 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1140317 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1686175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709568000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344610146 trainer/global_step‚199 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2003 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚55 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1829809 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16240766904832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49316352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1723858944 _step‚14 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1686175 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1387573760 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56148489809920 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345289728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56148489809920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16241184312320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1723858944 $ _timestamp‚1.784839589173068e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1140667 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 206569916928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288314368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1829809 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚822 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1839587 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9778 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56355085900288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205224627200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 254097082368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49239031808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚822 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3515984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚606 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15987087229952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1686181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680018944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚535508 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1686181 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚604813 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49239031808 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 205224627200 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1839587 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49239031808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1723858944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56148515983360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345289728 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3515984 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 254047766016 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56330804133146 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56353714437120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56332174171260 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9784 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 206224492220yÍã0ô+ _wandbruntime‚352  _runtime‚352߸ þõø .loss/FBt1/detection_loss‚5.160626860247566 %loss/total_loss‚36.51819307613846 trainer/global_step‚199 1loss/FBt2/assignment_loss‚0.24534393714772035 /loss/FRt2/detection_loss‚1.5410818846296368 epoch‚0 % _timestamp‚1.7848395892063317e+09 /loss/FRt2/assignment_loss‚5.323010451347684 2loss/SRqqt2/assignment_loss‚2.2846021250655646 0loss/FRt1/detection_loss‚0.46311651500730594 0loss/SRqqt1/detection_loss‚3.784027417604603 _step‚15 /loss/FRt1/assignment_loss‚4.565232118414767 1loss/SRqqt1/assignment_loss‚2.060467849204789 0loss/SRqqt2/detection_loss‚1.103333217269284 .loss/FBt2/detection_loss‚9.715798728259976 _runtime‚ 352.563553252 1loss/FBt1/assignment_loss‚0.27155197193956393_£öü /loss/FRt2/detection_loss‚1.5410818846296368 epoch‚0 % _timestamp‚1.7848395892063317e+09 /loss/FRt2/assignment_loss‚5.323010451347684 2loss/SRqqt2/assignment_loss‚2.2846021250655646 0loss/FRt1/detection_loss‚0.46311651500730594 0loss/SRqqt1/detection_loss‚3.784027417604603 _step‚15 /loss/FRt1/assignment_loss‚4.565232118414767 1loss/SRqqt1/assignment_loss‚2.060467849204789 0loss/SRqqt2/detection_loss‚1.103333217269284 .loss/FBt2/detection_loss‚9.715798728259976 _runtime‚ 352.563553252 1loss/FBt1/assignment_loss‚0.27155197193956393 .loss/FBt1/detection_loss‚5.160626860247566 %loss/total_loss‚36.51819307613846 trainer/global_step‚199 1loss/FBt2/assignment_loss‚0.24534393714772035 màú+÷‚(:   ¬û‰ÓÒ–ë³ disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.sda.out‚11.0625 disk.rbd1.out‚ 1.966796875 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.373424530029297 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0% memory_percent‚45.083586374918625 network.sent‚241149 disk.drbd1673.in‚0 disk.dm-0.out‚19.26513671875 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1584.in‚0 cpu‚24.645885992250754 proc.cpu.threads‚17) disk./.usagePercent‚70.30193999021581 disk.drbd1965.in‚0# disk.nvme0n1.out‚19.26513671875) proc.memory.availableMB‚ 13496.2578125 disk.drbd1965.out‚0 disk.nvme0n1.in‚9.765625 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0" proc.memory.rssMB‚ 2795.1328125 disk.drbd1226.out‚0 disk.drbd2078.out‚0 disk.sda2.in‚0.015625 network.recv‚67870% disk./.usageGB‚1257.0700874328613 disk.drbd2118.out‚0 disk.sda2.out‚11.0625 disk.rbd1.in‚0 disk.sda.in‚0.015625 disk.dm-0.in‚9.765625 disk.drbd1552.out‚0r­Û¡žø‚(:“ ¬û‰ÓÀ°É³ gpu.0.gpu‚56.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚67.0 gpu.0.powerWatts‚303.178' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.05933333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m€Wù‚(:’ »û‰Óõßß³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96241760253906* gpu.0.memoryAllocatedBytes‚ 49789337600 gpu.0.temp‚67.0 gpu.0.powerWatts‚271.505' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.50166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0moÙ%ú‚(:š  »û‰ÓøË´ disk.drbd2111.in‚0# disk.nvme0n1.out‚19.53466796875 disk.sda.in‚0.015625 disk.dm-0.out‚19.53466796875$ memory_percent‚45.08260091145833) disk./.usagePercent‚70.30194105690501 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 cpu‚24.668040296871368 network.sent‚246883 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚11.5625 disk.drbd1356.out‚0 disk.rbd1.out‚ 2.2724609375# proc.memory.availableMB‚13496.5" proc.memory.rssMB‚ 2795.1328125 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0% disk./.usageGB‚1257.0701065063477 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚9.765625 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.sda.out‚11.5625 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.373424530029297 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 network.recv‚69264 disk.drbd1673.in‚0 disk.dm-0.in‚9.765625 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd1828.out‚0 disk.drbd1965.out‚0dâ¬û‚(:’ Êû‰Ó̸׳ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚66.0 gpu.0.powerWatts‚288.104' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.03466666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*êº0ü‚(:¥  Êû‰Ó•¡õ³ disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.015625 proc.cpu.threads‚17 disk.sda2.out‚ 12.203125 disk.drbd1289.out‚0 disk.rbd1.out‚ 2.2724609375 disk.drbd1285.out‚0# disk.nvme0n1.out‚19.82373046875 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.dm-0.out‚19.82373046875 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.sda1.out‚0 network.sent‚247259 disk.drbd1828.in‚0 cpu‚24.689248802464693 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.373424530029297 network.recv‚69570 disk.drbd1931.in‚0 disk.sda.out‚ 12.203125) proc.memory.availableMB‚ 13496.2421875 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0" proc.memory.rssMB‚ 2795.1328125% disk./.usageGB‚1257.0700874328613 disk.dm-0.in‚9.765625 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.015625 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0% memory_percent‚45.083649953206375 disk.nvme0n1.in‚9.765625 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.30193999021581 disk.drbd2078.out‚0·†“ý‚(:ˆ Ùû‰ÓÓ驳 gpu.0.gpu‚39.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚66.0 gpu.0.powerWatts‚301.494' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.498 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö½Ž9þ‚(:®  Ùû‰ÓêÉ̳# proc.memory.rssMB‚ 2795.13671875 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.015625 disk.nvme0n1.in‚ 10.54296875 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.373440424601238 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.dm-0.out‚19.99560546875 disk.drbd1965.out‚0 disk.rbd1.out‚ 2.3349609375 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 12.69921875 disk.rbd1.in‚0# disk.nvme0n1.out‚19.99560546875 proc.cpu.threads‚17) disk./.usagePercent‚70.30193999021581 disk.sda.out‚ 12.69921875 disk.drbd1931.out‚0 disk.sda2.in‚0.015625' proc.memory.availableMB‚ 13495.90625 network.sent‚297455 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 10.54296875 disk.drbd2078.out‚0 disk.drbd1226.out‚0 cpu‚24.709602134907666% disk./.usageGB‚1257.0700874328613 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0% memory_percent‚45.085016886393234 network.recv‚72088 disk.drbd1356.in‚0 disk.drbd2041.in‚0dWJw’ÿ‚(:‡ èû‰Ó‡ÀÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚65.0 gpu.0.powerWatts‚281.436' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ÞÀ84€‚(:©  èû‰Óàõù³ disk.drbd2118.out‚0 disk.sda.out‚ 13.1015625 disk.sda2.in‚0.015625 cpu‚24.727431464817307 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2795.13671875 disk.drbd1817.out‚0 disk.drbd1673.out‚0# disk.nvme0n1.out‚20.39501953125 disk.drbd1552.in‚0 disk.drbd1584.in‚0 network.sent‚297719 disk.sda1.in‚0 disk.rbd1.in‚0$ memory_percent‚45.08310953776042% proc.memory.availableMB‚ 13496.375 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 2.3349609375 disk.drbd1226.in‚0 disk.drbd1550.in‚0% disk./.usageGB‚1257.0698471069336 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.dm-0.in‚ 10.54296875 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 10.54296875 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.sda.in‚0.015625 disk.drbd1938.in‚0 network.recv‚72352 disk.drbd1673.in‚0 disk.sda2.out‚ 13.1015625 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.dm-0.out‚20.39501953125 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.373440424601238 disk.drbd2078.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.30192654993184õ+ž‚(:“ ÷û‰Ó׌ճ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.788' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.92933333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`¶æ! ‚‚(:¦  ÷û‰Ó¯¶ö³ network.recv‚72658 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 cpu‚24.745227781487205! proc.memory.rssMB‚ 2795.140625 disk.drbd2118.out‚0 disk.drbd2078.in‚0 dis…^yy&k.drbd1828.out‚0 disk.drbd1510.in‚0* proc.memory.availableMB‚13496.36328125* proc.memory.percent‚11.373456319173176 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0% disk./.usageGB‚1257.0718231201172 disk.sda.out‚ 13.484375 disk.sda2.in‚0.015625 disk.sda2.out‚ 13.484375 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 10.54296875 disk.sda.in‚0.015625 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2111.in‚0# disk.nvme0n1.out‚20.53564453125 network.sent‚298025( disk./.usagePercent‚70.3020370589334 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.dm-0.out‚20.53564453125 disk.drbd1226.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0$ memory_percent‚45.08315722147624 disk.rbd1.out‚ 2.4609375 disk.nvme0n1.in‚ 10.54296875 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1828.in‚0@ìý£0ƒ+ _wandbruntime‚437  _runtime‚437ÛÎyÄ„¾ trainer/global_step‚249 % _timestamp‚1.7848396745514905e+09 _runtime‚ 437.908759505 _step‚16 & lr-AdamW/pg1‚0.0001041611669169975 & lr-AdamW/pg2‚0.0001041611669169975¥ ô†È… & lr-AdamW/pg1‚0.0001041611669169975 & lr-AdamW/pg2‚0.0001041611669169975 trainer/global_step‚249 % _timestamp‚1.7848396745514905e+09 _runtime‚ 437.908759505 _step‚16ðÌ´0†+ _wandbruntime‚437  _runtime‚437ÆãÂ'‡¼O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11999 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚638 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2286443 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 256736957952 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345282560 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 254962963242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1725956096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2004 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11999 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2096158 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344610034 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49241128960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680018944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70072013696512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709568000 % _timestamp‚1.7848396745526338e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4372837 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69815250565120 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1118855168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288314368 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2096164 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2286443 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚823 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69815276738560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 255391675392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20152932396032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚664766 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚751188 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19837656591360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667564503 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70070642240512 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 256307573276 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9764 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1416314 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417414656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70070642240512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49241128960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 255391675392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69815276738560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49241128960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371456000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1391768064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2096158 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 315325128192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69787236594064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚751544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚664770 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20153349810688 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 315275804672 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70043544167340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1725956096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709568000 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9770 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2276679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49323520 _step‚17 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371456000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345282560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚606 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70042174129338 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚249 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1415954 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370038002 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9764 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2276679 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19838024682496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9764 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2276679 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19838024682496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9770 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70070642240512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49241128960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 255391675392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69815276738560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49241128960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371456000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288314368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1391768064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2096158 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 315325128192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69787236594064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚751544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚664770 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20153349810688 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 315275804672 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70043544167340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1725956096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709568000 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9770 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2276679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49323520 _step‚17 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371456000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0±–àÿ0‰+ _wandbruntime‚439  _runtime‚439h7™Ê&ŠÄM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2004 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚667408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679330798 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2104525 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400070 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2104531 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19916247151616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1725956096 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2285761 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344610146 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1725956096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371463168 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 257759473152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11999 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49241128960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚823 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚754115 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20233187454464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427893248 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70323366742868 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70094164049920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 256414183424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70066012939872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 257759473152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1118855168 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2295539 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680018944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400070 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2104531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390286 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚823 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4390286 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚754470 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49241128960 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11999 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1421519 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚55 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70321996704754tÆ}5&‹ÈM VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70323366742868 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70094164049920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 256414183424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70066012939872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 257759473152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1118855168 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2295539 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680018944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400070 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2104531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390286 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667564503 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚823 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4390286 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680018944 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚754470 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49241128960 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11999 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1421519 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚55 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70321996704754 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2004 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚667408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679330798 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2104525 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370038114 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371463168 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400070 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2104531 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19916247151616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1725956096 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2285761 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344610146 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1725956096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371463168 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 257759473152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11999 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49241128960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚823 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚754115 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20233187454464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427893248¶ è0Œ+ _wandbruntime‚439  _runtime‚439g(0ûõ _step‚19 /loss/FRt2/detection_loss‚1.3818117385426754 &loss/total_loss‚34.884527318680824 epoch‚0 /loss/FRt2/assignment_loss‚5.333229746137347 0loss/FBt1/assignment_loss‚0.2834411791233887 trainer/global_step‚249 % _timestamp‚1.7848396760666797e+09 1loss/SRqqt2/assignment_loss‚2.231572994819054 /loss/SRqqt1/detection_loss‚2.99246544274091 .loss/FBt2/detection_loss‚7.134454348704356 /loss/FRt1/assignment_loss‚4.577633370249175 2loss/SRqqt1/assignment_loss‚1.5685113671444526 0loss/FBt2/assignment_loss‚0.1690817565650907 0loss/FRt1/detection_loss‚0.41172053843088285 1loss/SRqqt2/detection_loss‚1.2601237363688917 ,loss/FBt1/detection_loss‚7.5404810998546 _runtime‚ 439.423940848,{= ÿŽù 0loss/FBt2/assignment_loss‚0.1690817565650907 0loss/FRt1/detection_loss‚0.41172053843088285 1loss/SRqqt2/detection_loss‚1.2601237363688917 ,loss/FBt1/detection_loss‚7.5404810998546 _runtime‚ 439.423940848 _step‚19 /loss/FRt2/detection_loss‚1.3818117385426754 &loss/total_loss‚34.884527318680824 epoch‚0 /loss/FRt2/assignment_loss‚5.333229746137347 0loss/FBt1/assignment_loss‚0.2834411791233887 trainer/global_step‚249 % _timestamp‚1.7848396760666797e+09 1loss/SRqqt2/assignment_loss‚2.231572994819054 /loss/SRqqt1/detection_loss‚2.99246544274091 .loss/FBt2/detection_loss‚7.134454348704356 /loss/FRt1/assignment_loss‚4.577633370249175 2loss/SRqqt1/assignment_loss‚1.5685113671444526¶YÿÁ‚(:’ †ü‰Ó¼àij gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚61.0 gpu.0.powerWatts‚110.654' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.88466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÚ©5‚(:ª  †ü‰Ó‡šæ³% disk./.usageGB‚1257.0718154907227 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 10.54296875# disk.nvme0n1.out‚20.93017578125 disk.sda1.in‚0 disk.sda2.in‚0.015625 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.30203663225771 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0* proc.memory.availableMB‚13495.93359375 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 14.11328125 disk.sda2.out‚ 14.11328125 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0! proc.memory.rssMB‚ 2795.140625 network.recv‚74155 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.dm-0.out‚20.93017578125 disk.drbd1550.in‚0 disk.drbd1938.in‚0 network.sent‚303773 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 10.54296875 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0 cpu‚24.76038022256831 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 2.4609375 disk.drbd1584.in‚0$ memory_percent‚45.08490562438965* proc.memory.percent‚11.373456319173176 proc.cpu.threads‚17 disk.drbd2078.out‚0«)w2ž‘‚(:“ •ü‰Óö‘̳ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.322' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.10733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_FZR3’‚(:¨  •ü‰Óûí³ disk.drbd1584.out‚0 disk.sda2.out‚ 14.59765625 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 12.0546875! proc.memory.rssMB‚ 2795.140625 network.recv‚74503 disk.sda.out‚ 14.59765625 disk.dm-0.in‚ 12.0546875 cpu‚24.776607810281895 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0$ memory_percent‚45.08500099182129% disk./.usageGB‚1257.0718040466309 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 2.5234375 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0* proc.memory.availableMB‚13495.91015625( disk./.usagePercent‚70.3020359922442 disk.drbd2041.out‚0 disk.drbd1285.out‚0# disk.nvme0n1.out‚21.15087890625 disk.drbd1552.out‚0 disk.sda.in‚0.015625 disk.sda2.in‚0.015625 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.373456319173176 network.sent‚304121 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.dm-0.out‚21.15087890625 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0M7fž“‚(:“ ¤ü‰Ó¨¸½³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.512' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.17066666666668 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02‹Å*”‚(:Ÿ  ¤ü‰Óõ˜Ú³ disk.drbd1550.out‚0 disk.drbd1552.in‚0( disk./.usagePercent‚70.3020359922442 disk.dm-0.in‚ 12.0546875 disk.drbd2111.in‚0# disk.nvme0n1.out‚21.92236328125 disk.sda1.out‚0 disk.drbd1584.in‚0 cpu‚24.79044789642528 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1817.out‚0 disk.sda.out‚14.90625 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 network.sent‚304385 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0! proc.memory.rssMB‚ 2795.140625 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 network.recv‚74767 disk.drbd1289.out‚0 disk.drbd2111.out‚0' proc.memory.availableMB‚ 13495.84375 disk.sda.in‚0.015625 disk.dm-0.out‚21.92236328125 disk.drbd1356.in‚0 disk.sda2.out‚14.90625 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.373456319173176 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 2.5234375 disk.sda1.in‚0% memory_percent‚45.085271199544266 disk.nvme0n1.in‚ 12.0546875% disk./.usageGB‚1257.0718040466309 disk.sda2.in‚0.015625 disk.drbd1289.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0/’_³,•‚(:¡  ³ü‰Óüûî³ disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 15.203125 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 proc.memory.rssMB‚2795.25 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 13495.6328125( proc.memory.percent‚11.3739013671875 network.sent‚353849 disk.rbd1.in‚0 disk.nvme0n1.in‚ 12.0546875 disk.sda.in‚0.015625 disk.drbd1931.out‚0 disk.drbd1938.out‚0 cpu‚24.817186678228946# disk.nvme0n1.out‚22.56689453125 disk.drbd1510.out‚0 network.recv‚77480 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 15.203125 disk.dm-0.out‚22.56689453125 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 12.0546875 disk.drbd1356.in‚0 disk.drbd1356.out‚0$ memory_percent‚45.08612950642903 proc.cpu.threads‚17) disk./.usagePercent‚70.30203492555499 disk.rbd1.out‚ 2.6494140625 disk.sda1.in‚0% disk./.usageGB‚1257.0717849731445°†éž–‚(:“ ³ü‰ÓÅöÖ³ gpu.0.gpu‚58.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.894' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.63133333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê^vY6—‚(:«  Âü‰ÓýÙî³ disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 15.7578125 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0% memory_percent‚45.386648178100586# disk.nvme0n1.out‚22.89892578125 disk.drbd1226.in‚0* proc.memory.percent‚11.675580342610676 disk.sda.in‚0.015625 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 2.6494140625 disk.nvme0n1.in‚ 12.0546875 disk.drbd1550.out‚0 proc.cpu.threads‚17 network.sent‚383758) disk./.usagePercent‚70.30203492555499 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 12.0546875 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1965.in‚0 network.recv‚79856 disk.drbd2118.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0* proc.memory.availableMB‚13421.77734375! proc.memory.rssMB‚ 2869.390625 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.dm-0.out‚22.89892578125 disk.drbd1817.in‚0 disk.sda.out‚ 15.7578125% disk./.usageGB‚1257.0717849731445 disk.drbd2078.out‚0 disk.drbd2111.in‚0 cpu‚24.900409785494933 disk.drbd1584.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0áeMŽ˜‚(:ƒ Âü‰Ó¢àó´ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚58.0 gpu.0.powerWatts‚106.32' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.44 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÇ3$J™bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPX´!/JšbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPØ 7N›bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPŠð¯_JœbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÀk5rHbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPC{®‚NžbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPWQOHŸbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÃ¥îJ bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP*°qJ¡bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP˜a‡QN¢bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPªÊësH£bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPUUTN¤bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP÷fEbJ¥bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPüï,fN¦bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÀEÎ5J§bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPG$”bN¨bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPð‡ç§J©bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP·h6HªbC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP¼ ŸH«bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÌf;N¬bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPˆj'²J­bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPüsšEJ®bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP‰y·åH¯bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP=¡óJ°bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP·ýflH±bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP}žK9H²bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPê8Á—J³bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPwL÷*N£bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÚ—¸N¤bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPŸÓ‹N¥bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPXøÄcH¦bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPCg;§b6 jet/accuracy_3_of_4"trainer/global_step2:JP:© H¨bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPŒT(N©bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP-–â/CªbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:ã„¡<JPù2øJ«bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÏÎ=£J¬bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÇÉ€DN­bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPcßòJ®bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP5¨o3J¯bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÛ4›H°bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP-LH±bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÎàwwH²bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP2·>N³bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPžrH´bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPf";HµbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP7tk|J¶bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¥í}BN·bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPM_,N¸bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPhy¦J¹bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPP StJºbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPþ¦H»bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP Ú½9N¼bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP[„îJ½bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPE6ÇTJ¾bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP~>eI;¿b6 jet/accuracy_1_of_3"trainer/global_step2:JP—-«FHÀbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPR·HÁbC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¥‘u×@Âb; particle/accuracy_2_of_4"trainer/global_step2:JPmÀ"JJÃbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPzüǹNÄbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPX`ŽHÅbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPë— îNÆbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP»sò…JÇbE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP5èûHÈbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP>lÓzHÉbC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPïg¾ÌHÊbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP–'ŒJËbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPUuõÛHÌbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPNI„ñNÍbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP·âšVHÎbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPû¡+ÆJÏbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP÷ïNÐbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPçAâ\JÑbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP(Û©NÒbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP<ÍBHÓbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPazÀò<Ôb7 particle/sensitivity"trainer/global_step2:JP áœNÕbI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP:‰! @Öb; particle/accuracy_3_of_4"trainer/global_step2:JPš‡®eN×bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPU4+NØbI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP(¹ JÙbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPáóƒJÚbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPwµTÆNÛbI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPãklHÜbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPòÞBjNÝbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÆìåØJÞbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÙDð<;ßb6 jet/accuracy_3_of_3"trainer/global_step2:JP#JKHàbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPµýzJábE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPZ³Ké;âb6 jet/accuracy_4_of_4"trainer/global_step2:JP ýŽZJãbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPPóHäbC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPìÀJåbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP%.nJæbE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP±²Q„JçbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP æ=¾HèbC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPô§ÂuJébE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPV\Ö2JêbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‘m?;ëb6 jet/accuracy_2_of_2"trainer/global_step2:JPÜO§àHìbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPˆdßnHíbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP®U§ÔHîbC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPˆá«ÏJïbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÑv]HðbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPJ[Ä@ñb; particle/accuracy_2_of_3"trainer/global_step2:JP¢Ü‘HòbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPGœ“JóbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP³R-NôbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP‡Ü žJõbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPƒÿíNöbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP„=”gJ÷bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPJ2FÊJøbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPkéy^NùbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPb÷ä3NúbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPkiòJûbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP`϶JübE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP0z–ãHýbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPçi,¸JþbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPËϹÝHÿbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPCF‹IJ€bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPa™­ÆHbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÐý G‚bB validation_average_jet_accuracy"trainer/global_step2:JPØ1EXJƒbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP8?VˆH„bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPTS‡J…bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPòÐAåN†bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP+¹Y9‡b4 particle/accuracy"trainer/global_step2:JPç"SäHˆbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPŒ0¯H‰bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP©Žc6NŠbI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPwÊóH‹bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP`¹ðHŒbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPkÆ(qHbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP°$"NŽbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPj¯7<b7 particle/specificity"trainer/global_step2:JPz†JbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPïŸ%oN‘bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPô”*H’bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPe¶ÃèJ“bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPN¹&>J”bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPâŽô@•b; particle/accuracy_1_of_3"trainer/global_step2:JPƒºIN–bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP(BSN—bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÞõ¬&H˜bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP2ªÈŽJ™bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP蘖“HšbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP©¥íLN›bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP³)®ÌJœbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPLlS¿JbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¹ˆÇÂHžbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPõlÀŸJŸbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP´Bá; b6 jet/accuracy_2_of_3"trainer/global_step2:JPì°h£@¡b; particle/accuracy_1_of_2"trainer/global_step2:JP/ ø H¢bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP6ÞÍ’J£bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP @éH¤bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPK&AH¥bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP8C)ÅJ¦bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâ5 ùH§bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP —´H¨bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPt£H©bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPî~¼0ª+ _wandbruntime‚511  _runtime‚511Z?ùŒU:«Ït ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.012445888482034206 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.08269196003675461 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.05235143378376961 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.01164991408586502 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 (particle/accuracy‚0.7543479800224304 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.04268292710185051 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04453406110405922 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.03381776809692383 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922232031822205 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9236111640930176 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 *particle/specificity‚0.792752206325531 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.21089553833007812 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.027516020461916924 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.17360924184322357 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.0534304715692997 particle/accuracy_1_of_3‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.07992684096097946 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.125 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.032997261732816696 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03846153989434242 +jet/accuracy_2_of_3‚0.11288626492023468 particle/accuracy_1_of_2‚1 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.023052845150232315 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09736990928649902 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01591312326490879 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.0548643097281456 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0535714291036129 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.04994327574968338 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.04378896579146385 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0021008404437452555 9 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.057692307978868484 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05310880392789841 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.11363636702299118 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.1818181872367859 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982539713382721 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.05629190802574158 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996180534362793 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868708848953247 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ="Purity/1FRt1SRqqt*FBt/event_purity‚0.00022461813932750374 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5332687497138977 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.04363767430186272 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04383668303489685 & Purity/2FRt1SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9964522123336792 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8909187912940979 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971559643745422 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9902323484420776 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 )loss/val_total_loss‚36.94925032370562 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.01832330785691738 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.015684399753808975 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.05564894154667854 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.06384680420160294 ;"Purity/0FRt*SRqqt0FBt/event_purity‚0.039422303438186646 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.04378896579146385 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03846153989434242 epoch‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.052735425531864166 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0009920635493472219 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.016143936663866043 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.032328832894563675 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.02316862903535366 <"Purity/1FRt*SRqqt1FBt/event_purity‚0.0014285714132711291 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.09615384787321091 9 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.021969696506857872 _runtime‚ 511.145940131 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 trainer/global_step‚279 $ _timestamp‚1.784839747787838e+09 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.07765045017004013 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.04200548678636551 ;"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.059240102767944336 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0535714291036129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01568998582661152 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.012445888482034206 particle/accuracy_1_of_1‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.04615384712815285 particle/accuracy_1_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7358342409133911 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 _step‚20 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.19234584271907806 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.07774032652378082 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.07724694162607193 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9991883039474487 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0486111156642437 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 ;"Purity/0FRt1SRqqt*FBt/event_purity‚0.049136556684970856 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.06778910756111145 ;"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.052079204469919205 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.07721366733312607 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.011104141362011433 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 ;"Purity/*FRt*SRqqt*FBt/event_purity‚0.058301955461502075 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0486111156642437 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.19292663037776947 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.12308763712644577 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6179326772689819 ;"Purity/*FRt1SRqqt0FBt/event_purity‚0.024883201345801353 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ="Purity/1FRt1SRqqt0FBt/event_purity‚0.00023191094805952162 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 +jet/accuracy_1_of_4‚0.35054561495780945 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015310116112232208 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 /particle/accuracy_4_of_4‚0.9608165621757507 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9977837800979614 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9936729073524475 9 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.017673298716545105 ;"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.059785302728414536 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.053522396832704544 'particle/f_score‚0.5174135565757751 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.024227555841207504 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 +jet/accuracy_1_of_2‚0.20587658882141113 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06439264863729477 -jet/accuracy_3_of_4‚0.0051020411774516106 ( Purity/*FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.05151700973510742 ("Purity/0FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01591312326490879 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.028133932501077652 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16226115822792053 9 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.012445888482034206 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.00689117144793272 ("Purity/*FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ;"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.040444545447826385 ("Purity/0FRt2SRqqt*FBt/event_purity‚0 9 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.021801937371492386 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.06778910756111145 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.3625127673149109 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.025720985606312752 particle/accuracy_2_of_4‚1 ("Purity/*FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.020382395014166832 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.20300865173339844 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997063934803009 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912698864936829 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.04144560918211937 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0016917294124141335 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.02400909550487995 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06439264863729477 +particle/sensitivity‚0.6137652397155762 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 particle/accuracy_3_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.19899845123291016 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.011456792242825031 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.04974767938256264 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ;"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.050479378551244736 -jet/accuracy_3_of_3‚0.0013536119367927313 9 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.025881260633468628 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_4‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.1818181872367859 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.05773226171731949 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08625195920467377 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.04509371146559715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.0478113628923893 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.19669100642204285 ,jet/accuracy_2_of_2‚0.043102093040943146 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9548611640930176 9 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.028543082997202873 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99481600522995 <"Purity/1FRt0SRqqt1FBt/event_purity‚0.0020408162381500006 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.015118256211280823 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.761027991771698 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.003316326532512903 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6697973012924194 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.08028935641050339 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1mß$ƒÄ¬Ót =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.020382395014166832 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.20300865173339844 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997063934803009 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912698864936829 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.04144560918211937 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0016917294124141335 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.02400909550487995 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06439264863729477 +particle/sensitivity‚0.6137652397155762 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 particle/accuracy_3_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.19899845123291016 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.011456792242825031 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.04974767938256264 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ;"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.050479378551244736 -jet/accuracy_3_of_3‚0.0013536119367927313 9 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.025881260633468628 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_4‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.1818181872367859 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.05773226171731949 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08625195920467377 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.04509371146559715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.0478113628923893 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.19669100642204285 ,jet/accuracy_2_of_2‚0.043102093040943146 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9548611640930176 9 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.028543082997202873 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99481600522995 <"Purity/1FRt0SRqqt1FBt/event_purity‚0.0020408162381500006 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.015118256211280823 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.761027991771698 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.003316326532512903 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6697973012924194 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.08028935641050339 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.012445888482034206 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.08269196003675461 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.05235143378376961 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.01164991408586502 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 (particle/accuracy‚0.7543479800224304 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.04268292710185051 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04453406110405922 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.03381776809692383 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922232031822205 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9236111640930176 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 *particle/specificity‚0.792752206325531 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.21089553833007812 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.027516020461916924 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.17360924184322357 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.0534304715692997 particle/accuracy_1_of_3‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.07992684096097946 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.125 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.032997261732816696 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03846153989434242 +jet/accuracy_2_of_3‚0.11288626492023468 particle/accuracy_1_of_2‚1 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.023052845150232315 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09736990928649902 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01591312326490879 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.0548643097281456 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0535714291036129 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.04994327574968338 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.04378896579146385 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0021008404437452555 9 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.057692307978868484 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05310880392789841 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.11363636702299118 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.1818181872367859 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982539713382721 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.05629190802574158 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996180534362793 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868708848953247 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ="Purity/1FRt1SRqqt*FBt/event_purity‚0.00022461813932750374 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5332687497138977 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.04363767430186272 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04383668303489685 & Purity/2FRt1SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9964522123336792 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8909187912940979 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971559643745422 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9902323484420776 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 )loss/val_total_loss‚36.94925032370562 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.01832330785691738 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.015684399753808975 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.055648941r4ו54667854 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.06384680420160294 ;"Purity/0FRt*SRqqt0FBt/event_purity‚0.039422303438186646 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.04378896579146385 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03846153989434242 epoch‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.052735425531864166 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0009920635493472219 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.016143936663866043 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.032328832894563675 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.02316862903535366 <"Purity/1FRt*SRqqt1FBt/event_purity‚0.0014285714132711291 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.09615384787321091 9 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.021969696506857872 _runtime‚ 511.145940131 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 trainer/global_step‚279 $ _timestamp‚1.784839747787838e+09 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.07765045017004013 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.04200548678636551 ;"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.059240102767944336 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0535714291036129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01568998582661152 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.012445888482034206 particle/accuracy_1_of_1‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.04615384712815285 particle/accuracy_1_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7358342409133911 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 _step‚20 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.19234584271907806 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.07774032652378082 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.07724694162607193 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9991883039474487 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0486111156642437 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 ;"Purity/0FRt1SRqqt*FBt/event_purity‚0.049136556684970856 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.06778910756111145 ;"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.052079204469919205 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.07721366733312607 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.011104141362011433 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 ;"Purity/*FRt*SRqqt*FBt/event_purity‚0.058301955461502075 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0486111156642437 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.19292663037776947 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.12308763712644577 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6179326772689819 ;"Purity/*FRt1SRqqt0FBt/event_purity‚0.024883201345801353 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ="Purity/1FRt1SRqqt0FBt/event_purity‚0.00023191094805952162 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 +jet/accuracy_1_of_4‚0.35054561495780945 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015310116112232208 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 /particle/accuracy_4_of_4‚0.9608165621757507 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9977837800979614 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9936729073524475 9 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.017673298716545105 ;"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.059785302728414536 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.053522396832704544 'particle/f_score‚0.5174135565757751 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.024227555841207504 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 +jet/accuracy_1_of_2‚0.20587658882141113 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06439264863729477 -jet/accuracy_3_of_4‚0.0051020411774516106 ( Purity/*FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.05151700973510742 ("Purity/0FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01591312326490879 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.028133932501077652 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16226115822792053 9 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.012445888482034206 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.00689117144793272 ("Purity/*FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ;"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.040444545447826385 ("Purity/0FRt2SRqqt*FBt/event_purity‚0 9 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.021801937371492386 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.06778910756111145 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.3625127673149109 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.025720985606312752 particle/accuracy_2_of_4‚1 ("Purity/*FRt2SRqqt*FBt/event_purity‚0]sxm’­‚(:‡ Ñü‰ÓŒ™ã³ gpu.0.gpu‚10.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚61.0 gpu.0.powerWatts‚327.036' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚109.012 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>®(Û:®‚(:¯  Ñü‰ÓÞš‡´ disk.drbd2118.out‚0 cpu‚24.91559723334927 network.recv‚80120 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.sda2.in‚0.015625 disk.drbd1938.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.sda2.out‚ 16.30078125 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 11477.4609375 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.015625 disk.drbd1931.out‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.676979064941406 disk.sda1.in‚0 disk.drbd1289.in‚0! proc.memory.rssMB‚ 2869.734375 disk.drbd2111.out‚0 disk.rbd1.out‚ 2.7119140625 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 network.sent‚384022 disk.sda.out‚ 16.30078125 disk.dm-0.in‚ 12.0546875 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0% memory_percent‚53.298091888427734 disk.drbd2118.in‚0 disk.nvme0n1.in‚ 12.0546875 disk.drbd1510.out‚0) disk./.usagePercent‚70.40793625642813 disk.drbd1584.in‚0 disk.drbd1931.in‚0% disk.nvme0n1.out‚1962.20751953125% disk./.usageGB‚1258.9654083251953" disk.dm-0.out‚1962.207519531252˜ ¯‚(:’ àü‰Óʺ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚63.0 gpu.0.powerWatts‚275.707' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.90233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ºqî8°‚(:­  àü‰Ó×ÖÖ³ disk.drbd1552.out‚0 disk.sda2.in‚0.015625$ disk.nvme0n1.out‚1963.3583984375* proc.memory.percent‚11.676979064941406 disk.drbd1938.in‚0 disk.dm-0.in‚ 14.6953125 network.sent‚387184 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 network.recv‚81283 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0! disk.dm-0.out‚1963.3583984375$ memory_percent‚53.29904556274414 disk.drbd1965.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 16.65234375 disk.drbd1226.in‚0 disk.nvme0n1.in‚ 14.6953125 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.sda2.out‚ 16.65234375 disk.drbd2118.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 11477.2265625% disk./.usageGB‚1258.9654083251953 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1356.in‚0 cpu‚24.924376081583674 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.40793625642813 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.7119140625 disk.drbd1285.out‚0 disk.drbd1550.in‚0! proc.memory.rssMB‚ 2869.734375ò¼}Š0±+ _wandbruntime‚549  _runtime‚549q>åéƲÀ ' lr-AdamW/pg2‚0.00011712846879319192 trainer/global_step‚299 % _timestamp‚1.7848397858813047e+09 _runtime‚ 549.238561282 _step‚21 ' lr-AdamW/pg1‚0.000117128468793191923ÕsÊ³Ä ' lr-AdamW/pg2‚0.00011712846879319192 trainer/global_step‚299 % _timestamp‚1.7848397858813047e+09 _runtime‚ 549.238561282 _step‚21 ' lr-AdamW/pg1‚0.00011712846879319192Bœ‘0´+ _wandbruntime‚549  _runtime‚549{ü*Å'µ¿O XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86599346729569 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565747 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2603940 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 312199527936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23816472516608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86597976690323 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 311675684259 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709851648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚823 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371608576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24200783531520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49241128960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 549.242949908 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86321521838080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12222 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 384721985536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1729957 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚803595 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚823 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚823 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2779664 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1729643 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚926357 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1393865216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49241128960 _step‚22 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86635092974592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 312199527936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12297 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2603940 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 313020295537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚309 $ _timestamp‚1.784839785882382e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49241128960 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86633721366016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23816104425472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42879488qS¬É'¶ÃO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709851648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚823 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371608576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24200783531520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49241128960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 549.242949908 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86321521838080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12222 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 384721985536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1729957 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚803595 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚823 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚823 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2779664 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1729643 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚926357 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1393865216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49241128960 _step‚22 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86635092974592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 312199527936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12297 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2603940 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 313020295537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚309 $ _timestamp‚1.784839785882382e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49241128960 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86633721366016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23816104425472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42879488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86599346729569 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565747 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2603940 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 312199527936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23816472516608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86597976690323 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 311675684259“®ùý0·+ _wandbruntime‚550  _runtime‚550¤Ê† Á¸ÊM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2788746 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2612307 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚606 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86600457302016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _step‚23 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86600457302016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1725956096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 313222035968 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2788746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚309 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49241128960 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5411135 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚299 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2798822 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚929279 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚823 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 313222035968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385989659136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚823 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86915024780288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680171520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 312696485899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86879169305097 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86565128207808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86913653164544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 314567478272 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5411135 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24280637578240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86915024780288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2612313 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1735519 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1725956096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2612313 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 314567478272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 385946786816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚806235 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86877799265739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚314 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24281048541696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23895058882560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚823 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410963456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1735205 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚928970 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2004|™0vÔ&¹ÎM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚606 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86600457302016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _step‚23 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86600457302016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1725956096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 313222035968 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2788746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚309 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49241128960 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5411135 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚299 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2798822 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚929279 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚823 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 313222035968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385989659136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚823 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86915024780288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680171520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 312696485899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86879169305097 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86565128207808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86913653164544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 314567478272 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5411135 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24280637578240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86915024780288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2612313 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1735519 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1725956096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2612313 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 314567478272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 385946786816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚806235 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86877799265739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚314 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24281048541696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23895058882560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚823 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410963456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1735205 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚928970 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2004 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2788746 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2612307‰Ù?0º+ _wandbruntime‚550  _runtime‚550ºÄQþ»ø _runtime‚ 550.752754331 2loss/SRqqt1/assignment_loss‚1.6883387124732785 1loss/FBt2/assignment_loss‚0.13732972442760696 2loss/SRqqt2/assignment_loss‚2.1881964589714573 epoch‚1 /loss/FRt1/assignment_loss‚4.588369737648414 0loss/SRqqt1/detection_loss‚2.446119327308691 1loss/SRqqt2/detection_loss‚1.6180487195245592 /loss/FBt2/detection_loss‚14.006423551010672 %loss/total_loss‚37.19591160705967 trainer/global_step‚299 _step‚24 /loss/FRt2/assignment_loss‚5.329820883143079 0loss/FBt1/assignment_loss‚0.2113351218368583 0loss/FRt1/detection_loss‚0.42539417728611656 .loss/FRt2/detection_loss‚1.434103903004043 .loss/FBt1/detection_loss‚3.122431290424897 $ _timestamp‚1.784839787395529e+09€›Ý¼ü 1loss/FBt2/assignment_loss‚0.13732972442760696 2loss/SRqqt2/assignment_loss‚2.1881964589714573 epoch‚1 /loss/FRt1/assignment_loss‚4.588369737648414 0loss/SRqqt1/detection_loss‚2.446119327308691 1loss/SRqqt2/detection_loss‚1.6180487195245592 /loss/FBt2/detection_loss‚14.006423551010672 %loss/total_loss‚37.19591160705967 trainer/global_step‚299 _step‚24 /loss/FRt2/assignment_loss‚5.329820883143079 0loss/FBt1/assignment_loss‚0.2113351218368583 0loss/FRt1/detection_loss‚0.42539417728611656 .loss/FRt2/detection_loss‚1.434103903004043 .loss/FBt1/detection_loss‚3.122431290424897 $ _timestamp‚1.784839787395529e+09 _runtime‚ 550.752754331 2loss/SRqqt1/assignment_loss‚1.6883387124732785ÞìU:½‚(:¯  ïü‰Ó¹ÎÔ³* proc.memory.percent‚11.677058537801106 network.recv‚81547 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 14.6953125 disk.drbd2041.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 14.6953125 disk.drbd1552.out‚0 cpu‚24.933100225034295 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2118.in‚0$ memory_percent‚53.30052375793457 network.sent‚387448 disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd2078.out‚0! disk.dm-0.out‚1963.6279296875 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2869.75390625% disk./.usageGB‚1258.9536399841309 disk.rbd1.in‚0 disk.drbd1584.out‚0* proc.memory.availableMB‚11476.86328125 disk.drbd1584.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.sda.out‚ 16.85546875 disk.drbd1965.in‚0) disk./.usagePercent‚70.40727810918911 disk.sda.in‚0.015625 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0$ disk.nvme0n1.out‚1963.6279296875 disk.drbd1226.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 2.837890625 disk.drbd1817.in‚0 disk.sda2.out‚ 16.85546875÷¿v_’¾‚(:‡ ïü‰ÓÕ¹±³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚62.0 gpu.0.powerWatts‚296.796' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100226389694864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100631747055616 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 364734519296 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3235616 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28192854198784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100226415122832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2004 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100630375447040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709851648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚638 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1073883 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49241128960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 363389084160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12222 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2009836 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100267012536320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚823 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1725956096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371608576 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 363389084160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100631747055616 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 364734519296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 447431475712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49241128960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12297 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1074236 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100266986362880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27745790814208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚823 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28192854198784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100226415122832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2004 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100630375447040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709851648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚638 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1073883 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49241128960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 363389084160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12222 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2009836 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100267012536320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚823 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1725956096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371608576 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 363389084160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100631747055616 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 364734519296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 447431475712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49241128960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12297 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1074236 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100266986362880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27745790814208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚823 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100226389694864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100631747055616 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 364734519296 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3235616 HDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3030657 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 448729703424 trainer/global_step‚349 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6285437 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1076800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49241128960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚823 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6275355 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100545921826816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57552384 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 448672151040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100505191468640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 365144980097 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100545895653376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3244698 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚938240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2004 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3030657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49241128960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425643520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28272681240064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚823 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1393865216 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 363800368707 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100911678861312 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3030663 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3254774 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _step‚27 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27824377180160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100868991837347 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1725956096 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 364411592192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2015035 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2004 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100505216896608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49241128960 % _timestamp‚1.7848398738317251e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6285437 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100870361876705 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚823 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3030663 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680171520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6285437 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1076800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49241128960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚823 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6275355 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100545921826816 TEDeviceStatsMonitor.on_train_batch_end/ala ûP!located_bytes.small_pool.peak‚ 1680171520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57552384 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 448672151040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100505191468640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 365144980097 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100545895653376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚638 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3244698 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚938240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2004 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3030657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49241128960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425643520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28272681240064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚823 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1393865216 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 363800368707 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100911678861312 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3030663 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3254774 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _step‚27 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27824377180160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100868991837347 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1725956096 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 364411592192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2015035 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2004 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100505216896608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49241128960 % _timestamp‚1.7848398738317251e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6285437 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100870361876705 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚823 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3030663 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680171520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0\`³S0Ò+ _wandbruntime‚637  _runtime‚637sKŒgÓú 1loss/FBt2/assignment_loss‚0.19181329099774302 .loss/FBt2/detection_loss‚7.867600424176833 .loss/FRt2/assignment_loss‚4.57824163498417 0loss/FBt1/assignment_loss‚0.3320378898039046 0loss/FRt1/detection_loss‚0.38905118830478974 /loss/FRt2/detection_loss‚1.3425055056795359 /loss/FRt1/assignment_loss‚4.557297064759637 2loss/SRqqt2/assignment_loss‚1.6246743256627407 1loss/SRqqt1/detection_loss‚3.0425137115164893 /loss/FBt1/detection_loss‚2.8218914901722023 trainer/global_step‚349 $ _timestamp‚1.784839873866167e+09 _runtime‚ 637.223372715 2loss/SRqqt1/assignment_loss‚1.9266574603977689 1loss/SRqqt2/detection_loss‚1.1365154097879904 &loss/total_loss‚29.810799396243805 epoch‚1 _step‚28ü-dÔþ _runtime‚ 637.223372715 2loss/SRqqt1/assignment_loss‚1.9266574603977689 1loss/SRqqt2/detection_loss‚1.1365154097879904 &loss/total_loss‚29.810799396243805 epoch‚1 _step‚28 1loss/FBt2/assignment_loss‚0.19181329099774302 .loss/FBt2/detection_loss‚7.867600424176833 .loss/FRt2/assignment_loss‚4.57824163498417 0loss/FBt1/assignment_loss‚0.3320378898039046 0loss/FRt1/detection_loss‚0.38905118830478974 /loss/FRt2/detection_loss‚1.3425055056795359 /loss/FRt1/assignment_loss‚4.557297064759637 2loss/SRqqt2/assignment_loss‚1.6246743256627407 1loss/SRqqt1/detection_loss‚3.0425137115164893 /loss/FBt1/detection_loss‚2.8218914901722023 trainer/global_step‚349 $ _timestamp‚1.784839873866167e+09ócXƒ9Õ‚(:®  Éý‰Óé‘ð³ network.recv‚89544% disk.nvme0n1.out‚1966.64697265625 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 14.6953125 disk.sda1.in‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 19.41796875 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.sda2.in‚0.015625 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.40727725583774 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 3.0888671875" disk.dm-0.out‚1966.64697265625 network.sent‚512801 disk.dm-0.in‚ 14.6953125* proc.memory.percent‚11.677074432373047 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.drbd2078.out‚0 cpu‚24.978426894488308% disk./.usageGB‚1258.9536247253418 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd2041.in‚0' proc.memory.availableMB‚ 9951.109375" proc.memory.rssMB‚ 2869.7578125 disk.sda.out‚ 19.41796875 disk.drbd1965.out‚0$ memory_percent‚59.50883229573568UÔ­Ö‚(:’ Éý‰Ó¤œÁ³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚64.0 gpu.0.powerWatts‚283.147' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.38233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äö±Òׂ(:’ Øý‰Ó•ýȳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96485900878906* gpu.0.memoryAllocatedBytes‚ 49791434752 gpu.0.temp‚62.0 gpu.0.powerWatts‚277.292' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.43066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h·y\&Ø‚(:›  Øý‰Óÿºë³ disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0" disk.dm-0.out‚1967.00927734375 network.recv‚89718 disk.drbd1817.out‚0 disk.sda2.in‚0.015625 disk.nvme0n1.in‚ 14.6953125 cpu‚24.98394181987229 proc.cpu.threads‚17 disk.sda.out‚20 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.sda.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1226.in‚0& proc.memory.availableMB‚ 9658.21875 disk.sda2.out‚20 disk.drbd1965.out‚0 disk.rbd1.out‚ 3.2685546875% disk.nvme0n1.out‚1967.00927734375 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2869.76953125 disk.dm-0.in‚ 14.6953125 network.sent‚512975 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0) disk./.usagePercent‚70.40727725583774$ memory_percent‚60.70060729980469 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.677122116088867 disk.drbd1938.out‚0% disk./.usageGB‚1258.9536247253418 disk.drbd1828.out‚0 disk.drbd2111.in‚0\ڜق(:’ çý‰Ó†¶Â³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.1' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.03333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¢2Ú‚(:§  çý‰Óîùâ³ disk.drbd1552.in‚0 network.recv‚90656% disk./.usageGB‚1258.9536247253418 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.015625 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 3.5107421875 disk.drbd1584.out‚0 cpu‚24.99063697724277 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0" proc.memory.rssMB‚ 2869.7734375 disk.sda.in‚0.015625 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 20.578125 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0" disk.dm-0.out‚1967.54052734375 disk.nvme0n1.in‚ 15.609375 disk.drbd1289.in‚0 disk.dm-0.in‚ 15.609375 disk.drbd1226.out‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚1967.54052734375 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 network.sent‚518549 disk.drbd1817.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.677138010660808 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0$ memory_percent‚61.21374766031901) disk./.usagePercent‚70.40727725583774 disk.drbd2118.in‚0 disk.sda2.out‚ 20.578125 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0' proc.memory.availableMB‚ 9532.109375 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.drbd1285.in‚0²YÉÕ9Û‚(:®  öý‰Óõ‹ñ³ disk.drbd1828.out‚0% disk.nvme0n1.out‚1968.01611328125 disk.sda2.out‚ 20.97265625 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.40727832252695 disk.rbd1.out‚ 3.5107421875 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0" proc.memory.rssMB‚ 2869.7734375 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1550.in‚0$ memory_percent‚61.21382713317871) proc.memory.availableMB‚ 9532.08984375% disk./.usageGB‚1258.9536437988281 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.015625 cpu‚24.996696298900897 disk.sda.out‚ 20.97265625 disk.drbd1965.out‚0" disk.dm-0.out‚1968.01611328125 network.sent‚518681* proc.memory.percent‚11.677138010660808 network.recv‚90788 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 15.609375 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 15.609375 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 ›*)œÜ‚(:‘ öý‰Óܳճ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚297.721' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.24033333333333 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éˆipÝ‚(:’ …þ‰Ó‚»­³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.76' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.25333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÕ‰]7Þ‚(:¬  …þ‰Ó¿Íγ disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.015625 disk.drbd1285.in‚0 network.recv‚93789 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 15.609375 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 21.2890625% disk.nvme0n1.out‚1968.24267578125 disk.drbd1550.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0" disk.dm-0.out‚1968.24267578125 disk.drbd2041.out‚0 cpu‚25.001859402479056 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1673.out‚0 network.sent‚583550 disk.drbd1673.in‚0 disk.drbd1931.in‚0% memory_percent‚61.214288075764976 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.40727725583774 disk.sda.out‚ 21.2890625 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 3.5732421875 disk.drbd1550.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2869.7734375 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 9531.9765625* proc.memory.percent‚11.677138010660808% disk./.usageGB‚1258.9536247253418 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 15.609375çîÕ\žß‚(:“ ”þ‰Óúíâ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.335' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.11166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯*À™7à‚(:¬  ”þ‰ÓÃ…´ disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.40727725583774 disk.drbd2118.in‚0 disk.sda.out‚ 21.72265625% disk./.usageGB‚1258.9536247253418 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd2041.in‚0$ disk.nvme0n1.out‚1968.7470703125 network.recv‚94398 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0$ memory_percent‚61.21427218119303 disk.sda1.in‚0 disk.rbd1.out‚ 3.5732421875 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 network.sent‚583919 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.dm-0.in‚ 15.609375 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 15.609375 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 9531.98046875 cpu‚25.007434602594756 disk.drbd2111.in‚0! disk.dm-0.out‚1968.7470703125 disk.drbd1584.in‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 21.72265625 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.015625 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2869.7734375* proc.memory.percent‚11.677138010660808WÒ'0á+ _wandbruntime‚722  _runtime‚722¾±à|ÆâÀ ' lr-AdamW/pg1‚0.00011710728286079612 ' lr-AdamW/pg2‚0.00011710728286079612 trainer/global_step‚399 % _timestamp‚1.7848399593032675e+09 _runtime‚ 722.660500692 _step‚29ââQ<ÊãÄ trainer/global_step‚399 % _timestamp‚1.7848399593032675e+09 _runtime‚ 722.660500692 _step‚29 ' lr-AdamW/pg1‚0.00011710728286079612 ' lr-AdamW/pg2‚0.00011710728286079612ç<>00ä+ _wandbruntime‚722  _runtime‚722‹þ`'åÐO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3691568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚379 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1067600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114581731872785 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611278 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10068 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1067595 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114627029528064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114212450887680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3701630 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12297 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427747840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3440646 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114628401136640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2288151 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 509948916224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚ 722.661822716 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3440646 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚30 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3701630 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 415924075520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32185058028032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32184630280192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12297 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 415228061153 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚399 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114212477061120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114580361833539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31675109111808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12222 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59656704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114212450887680 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3440640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10062 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345435136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31674741020672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7132208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114628401136640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31674741020672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7132208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114628401136640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1067595 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114627029528064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114212450887680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3701630 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12297 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427747840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3440646 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114628401136640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2288151 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 509948916224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚ 722.661822716 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3440646 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚30 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3701630 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 415924075520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32185058028032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32184630280192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12297 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 415228061153 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚399 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114212477061120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114580361833539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31675109111808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12222 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59656704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0n ÿ0ç+ _wandbruntime‚724  _runtime‚724/Ä ß&èÙM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114491360178176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 $ _timestamp‚1.784839960781717e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2293327 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1070235 _runtime‚ 724.139368785 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3449013 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3700650 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3710726 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1223092 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31753695477760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114908332942336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2293705 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1070240 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7159739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 414904251515 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3449007 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114860184408955 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31753327386624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 trainer/global_step‚399 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3710726 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114908332942336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680171520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61746688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114491386351616 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 416946590720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚31 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3449007 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3700650 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 511195112960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 416248862905 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7159739 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429837824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32264890590720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114491386351616 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114491360178176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7149657 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 416946590720 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 415601148416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114445305585408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824²ð…!ã&éÝM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1223465 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1223092 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31753695477760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114908332942336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2293705 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1070240 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7159739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 414904251515 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3449007 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114860184408955 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31753327386624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 trainer/global_step‚399 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3710726 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114908332942336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680171520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61746688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114491386351616 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 416946590720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚31 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3449007 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3700650 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 511195112960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 416248862905 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7159739 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429837824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32264890590720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114491386351616 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114491360178176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7149657 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 416946590720 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 415601148416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114445305585408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114491360178176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 $ _timestamp‚1.784839960781717e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2293327 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1070235 _runtime‚ 724.139368785 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3449013 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3700650 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3710726 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚376 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 572299517952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345435136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427747840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10062 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4147520 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2567600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚371 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 466331943961 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8006510 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565747 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128157941585920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3858996 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1367629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128106567072464 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3858990 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4157582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚128157941585920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚449 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3858996 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 465768196608 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 465768196608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36176299179520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59656704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.784840046092363e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128623683609088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128157915412480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680171520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚128572924444393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35604427409408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 467113631744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35604059318272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611278 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚376 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 572299517952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345435136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427747840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10062 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4147520 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2567600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚371 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 466331943961 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8006510 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565747 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128157941585920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3858996 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1367629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128106567072464 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3858990 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4157582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚128157941585920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚449 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3858996 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 465768196608 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 465768196608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36176299179520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59656704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.784840046092363e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128623683609088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128157915412480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680171520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚128572924444393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35604427409408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 467113631744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35604059318272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611278"ÖB–ë”ý‚(:‰ îþ‰Óë̾³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.597' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.199 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04µeŽ5þ‚(:ª  îþ‰ÓóîÞ³ disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0$ disk.nvme0n1.out‚1972.8955078125 disk.sda.out‚ 23.96484375 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0$ memory_percent‚61.21068000793457 proc.cpu.threads‚17 disk.dm-0.in‚18.09375 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0$ disk./.usageGB‚1258.953441619873 disk.drbd1552.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.677153905232746 disk.sda2.in‚0.015625 disk.sda2.out‚ 23.96484375 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 9532.86328125 disk.drbd1552.out‚0 disk.sda.in‚0.015625 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0! disk.dm-0.out‚1972.8955078125 disk.drbd2041.out‚0 disk.nvme0n1.in‚18.09375 network.recv‚98835 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1828.in‚0 cpu‚25.035995489772798 network.sent‚652064 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2869.77734375 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 3.8876953125 disk.drbd1285.in‚0) disk./.usagePercent‚70.407267015621370nÍ0ÿ+ _wandbruntime‚810  _runtime‚810ÏeÊîÓ€ÙM J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2572770 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1370899 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 468136146944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 F:DeviceStatsMonitor.on_train_batch_end/active.all.alloàÖû %cated‚8034041 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35683013775360 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128851376980563 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4166678 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4156602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128904987023360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8034041 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2573139 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 573487488512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128385368846240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1370535 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35682645684224 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3867357 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128385394274208 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128904987023360 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8023959 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚364 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36256554621952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128436824702976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4166678 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128436850876416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8023959 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53358080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3867363 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36256133172736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 466008134323 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚449 $ _timestamp‚1.784840047570182e+09 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1202240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 810.927817276fÊ›ã&ÝM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1370535 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35682645684224 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3867357 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128385394274208 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128904987023360 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8023959 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚364 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36256554621952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128436824702976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4166678 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128436850876416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8023959 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53358080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3867363 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36256133172736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 466008134323 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚449 $ _timestamp‚1.784840047570182e+09 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1202240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 810.927817276 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2572770 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1370899 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 468136146944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8034041 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35683013775360 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128851376980563 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4166678 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4156602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128904987023360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8034041 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2573139 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 573487488512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128385368846240#ôꩃ0‚+ _wandbruntime‚810  _runtime‚810 È™Fƒû epoch‚1 1loss/SRqqt2/assignment_loss‚1.602029210470181 .loss/FBt1/detection_loss‚9.929036975932503 % _timestamp‚1.7848400476037607e+09 1loss/SRqqt1/detection_loss‚3.3887215860956377 /loss/FBt2/detection_loss‚28.865491410162935 trainer/global_step‚449 _runtime‚ 810.960987557 _step‚36 2loss/SRqqt1/assignment_loss‚1.8416678780789366 1loss/FBt1/assignment_loss‚0.19390931596196742 0loss/FBt2/assignment_loss‚0.2630497455942725 1loss/SRqqt2/detection_loss‚1.0506639459799945 &loss/total_loss‚57.452396150208784 /loss/FRt1/assignment_loss‚4.602902014297465 0loss/FRt2/assignment_loss‚3.8565030678344137 /loss/FRt1/detection_loss‚0.4188287248284496 /loss/FRt2/detection_loss‚1.4395922749720294H€Ñ„ÿ /loss/FBt2/detection_loss‚28.865491410162935 trainer/global_step‚449 _runtime‚ 810.960987557 _step‚36 2loss/SRqqt1/assignment_loss‚1.8416678780789366 1loss/FBt1/assignment_loss‚0.19390931596196742 0loss/FBt2/assignment_loss‚0.2630497455942725 1loss/SRqqt2/detection_loss‚1.0506639459799945 &loss/total_loss‚57.452396150208784 /loss/FRt1/assignment_loss‚4.602902014297465 0loss/FRt2/assignment_loss‚3.8565030678344137 /loss/FRt1/detection_loss‚0.4188287248284496 /loss/FRt2/detection_loss‚1.4395922749720294 epoch‚1 1loss/SRqqt2/assignment_loss‚1.602029210470181 .loss/FBt1/detection_loss‚9.929036975932503 % _timestamp‚1.7848400476037607e+09 1loss/SRqqt1/detection_loss‚3.3887215860956377$ƒÐësŸ…‚(:” ýþ‰ÓÚ­®³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.954' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.98466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04YæS2†‚(:§  ýþ‰Ó¡øɳ network.recv‚101402! disk.dm-0.out‚1973.2431640625 disk.nvme0n1.in‚18.09375 disk.drbd1226.out‚0 disk.drbd1817.out‚0# memory_percent‚61.2119197845459 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0$ disk.nvme0n1.out‚1973.2431640625 disk.drbd1938.out‚0 disk.sda.in‚0.015625) disk./.usagePercent‚70.40727149571605% disk./.usageGB‚1258.9535217285156 disk.drbd1550.out‚0 disk.drbd1817.in‚0) proc.memory.availableMB‚ 9532.55859375 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 network.sent‚714241 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2869.77734375 cpu‚25.040035389742886 disk.sda2.out‚ 24.546875 disk.drbd2118.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.677153905232746 disk.drbd2041.out‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚18.09375 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.sda.out‚ 24.546875 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 3.9501953125 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0¼_ûž‡‚(:“ Œÿ‰Ó˜ôä³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.929' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.64299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Ú–2ˆ‚(:§  Œÿ‰Ó±µƒ´ disk.drbd1510.in‚0 disk.drbd1817.out‚0" disk.dm-0.out‚1974.68115234375 disk.drbd1965.out‚0 disk.sda.out‚24.84375 disk.drbd1828.out‚0 disk.rbd1.out‚ 3.9501953125 disk.drbd1510.out‚0 disk.sda2.out‚24.84375 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625) proc.memory.availableMB‚ 9532.55859375 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2869.77734375 disk.nvme0n1.in‚18.09375 disk.sda.in‚0.015625* proc.memory.percent‚11.677153905232746 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 network.sent‚714307 network.recv‚101468% disk./.usageGB‚1258.9535217285156 disk.drbd1584.in‚0 cpu‚25.044185936572042 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0# memory_percent‚61.2119197845459) disk./.usagePercent‚70.40727149571605 disk.drbd1938.out‚0 disk.dm-0.in‚18.09375 disk.drbd1356.out‚0 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0% disk.nvme0n1.out‚1974.68115234375 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0]fY±œ‰‚(:‘ ›ÿ‰Óªì½³ gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚107.816' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.93866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bž[6Š‚(:«  ›ÿ‰Ó êÚ³ disk.sda2.in‚0.015625 disk.drbd1289.out‚0) proc.memory.availableMB‚ 9532.55859375 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0# memory_percent‚61.2119197845459" disk.dm-0.out‚1975.09521484375% disk.nvme0n1.out‚1975.09521484375 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1226.in‚0 network.recv‚101576 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚18.09375 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 25.08203125 disk.sda.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.40727042902684 disk.drbd2111.out‚0 disk.drbd1817.out‚0 network.sent‚714415 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 25.08203125 disk.drbd1673.out‚0* proc.memory.percent‚11.677153905232746 disk.drbd1356.in‚0 cpu‚25.04795996182661 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2869.77734375% disk./.usageGB‚1258.9535026550293 disk.dm-0.in‚18.09375 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.076171875ïD}1Ÿ‹‚(:” ªÿ‰ÓÝÉdz gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.893' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.96433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„+Q²5Œ‚(:ª  ªÿ‰Óåê³ disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 25.80078125 disk.dm-0.in‚18.09375 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0) proc.memory.availableMB‚ 9532.55859375 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.sda.in‚0.015625 cpu‚25.051583970586062 disk.drbd1931.out‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.40726829564844 disk.rbd1.out‚ 4.076171875 disk.drbd1552.out‚0# memory_percent‚61.2119197845459 disk.sda2.in‚0.015625 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚18.09375$ disk.nvme0n1.out‚1976.7138671875 proc.cpu.threads‚17 network.recv‚101684 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0% disk./.usageGB‚1258.9534645080566 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2869.77734375 disk.drbd1931.in‚0! disk.dm-0.out‚1976.7138671875 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 25.80078125 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.677153905232746 network.sent‚714523w¡Ÿr2‚(:§  ¹ÿ‰ÓÎßé³ cpu‚25.055095740838453 disk.sda2.out‚ 26.3828125 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 network.sent‚720016 disk.drbd1828.in‚0) disk./.usagePercent‚70.40726573559434 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 26.3828125 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1552.out‚0 network.recv‚102739% disk./.usageGB‚1258.9534187316895 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 4.138671875 disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 9532.5390625$ disk.nvme0n1.out‚1976.9052734375 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.015625 disk.drbd1828.out‚0! disk.dm-0.out‚1976.9052734375 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2869.77734375 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚18.09375 disk.drbd2111.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.677153905232746 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0# memory_percent‚61.2119992574056 disk.drbd1584.in‚0 disk.nvme0n1.in‚18.09375æJ“ÝžŽ‚(:“ ¹ÿ‰ÓȪ˳ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.048' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.68266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0roÒH0+ _wandbruntime‚896  _runtime‚8965\xÄÆÀ ' lr-AdamW/pg1‚0.00011705540347006973 ' lr-AdamW/pg2‚0.00011705540347006973 trainer/global_step‚499 % _timestamp‚1.7848401329942787e+09 _runtime‚ 896.351531418 _step‚37wBÅDÊ‘Ä _step‚37 ' lr-AdamW/pg1‚0.00011705540347006973 ' lr-AdamW/pg2‚0.00011705540347006973 trainer/global_step‚499 % _timestamp‚1.7848401329942787e+09 _runtime‚ 896.351531418%}±1ƒ0’+ _wandbruntime‚896  _runtime‚896`æDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D:DeviceStatsMonitor.on_train_batc%C¼ÅŽ#h_start/active.all.current‚10068 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12297 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4277340 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142621709298688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4613534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4603472 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142620337690112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8890880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142562746976755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1514929 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634647476736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680171520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 516091215491 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40167971727360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2846524 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 517435826769 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371608576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709851648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2846893 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 518303187968 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4603472 trainer/global_step‚499 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 516957752832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611278 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824ÍIìhÚ'”ÔO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39533745707008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 516957752832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142103406110720 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142103379937280 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4277346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142103406110720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12222 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8880812 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 518303187968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10062 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142046681189232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1331595 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142564117016001 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565747 % _timestamp‚1.7848401329953747e+09 _runtime‚ 896.354522872 _step‚38 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4613534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39533377615872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2846893 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 518303187968 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4603472 trainer/global_step‚499 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 516957752832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611278 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421456384 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634594111488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12297 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53365248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10068 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12297 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4277340 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142621709298688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680171520 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4613534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4603472 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142620337690112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8890880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142562746976755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1514929 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634647476736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680171520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 516091215491 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40167971727360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2846524 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 517435826769 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371608576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709851648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39612332072960 _step‚39 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2852423 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4622630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 trainer/global_step‚499 % _timestamp‚1.7848401344764972e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4622630 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142325482963008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2852062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 _runtime‚ 897.834158355 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8908343 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4612554 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 519325703168 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421449216 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 635895202304 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8898261 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 517980260864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1518183 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142901641104384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142382315401216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8898261 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142900269488640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4285713 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 517980260864 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8908343 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1517827 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39611963981824 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2852062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 _runtime‚ 897.834158355 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8908343 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4612554 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 519325703168 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421449216 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 635895202304 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8898261 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 517980260864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1518183 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142901641104384 [GDeviceStayÂð«tsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142382315401216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8898261 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142900269488640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12222 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4285713 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 517980260864 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8908343 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1517827 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39611963981824 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39612332072960 _step‚39 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2852423 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4622630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039358 trainer/global_step‚499 % _timestamp‚1.7848401344764972e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4622630 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142325482963008'Ó@ 0˜+ _wandbruntime‚897  _runtime‚897ø€K„ý™÷ /loss/FRt2/assignment_loss‚3.616881474852562 1loss/FBt1/assignment_loss‚0.23769922605285956 .loss/FRt2/detection_loss‚1.376207078862608 1loss/SRqqt2/detection_loss‚1.0126038028625466 epoch‚1 1loss/SRqqt1/assignment_loss‚1.951595336523923 0loss/SRqqt1/detection_loss‚3.008027839729233 .loss/FBt2/detection_loss‚7.789987414840185 _runtime‚ 897.866569464 /loss/FRt1/assignment_loss‚4.559893667779893 1loss/SRqqt2/assignment_loss‚1.496252882601329 1loss/FBt2/assignment_loss‚0.17852589074333283 .loss/FBt1/detection_loss‚6.770709109310803 &loss/total_loss‚32.378623644403945 _step‚40 /loss/FRt1/detection_loss‚0.3802399202446705 trainer/global_step‚499 % _timestamp‚1.7848401345093348e+09Ÿšû 1loss/FBt2/assignment_loss‚0.17852589074333283 .loss/FBt1/detection_loss‚6.770709109310803 &loss/total_loss‚32.378623644403945 _step‚40 /loss/FRt1/detection_loss‚0.3802399202446705 trainer/global_step‚499 % _timestamp‚1.7848401345093348e+09 /loss/FRt2/assignment_loss‚3.616881474852562 1loss/FBt1/assignment_loss‚0.23769922605285956 .loss/FRt2/detection_loss‚1.376207078862608 1loss/SRqqt2/detection_loss‚1.0126038028625466 epoch‚1 1loss/SRqqt1/assignment_loss‚1.951595336523923 0loss/SRqqt1/detection_loss‚3.008027839729233 .loss/FBt2/detection_loss‚7.789987414840185 _runtime‚ 897.866569464 /loss/FRt1/assignment_loss‚4.559893667779893 1loss/SRqqt2/assignment_loss‚1.496252882601329(\кž›‚(:“ Èÿ‰Óè¾å³ gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.934' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.31133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùwЀ4œ‚(:©  Èÿ‰ÓöЉ´ disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0) proc.memory.availableMB‚ 9533.44140625 disk.drbd1938.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 26.6953125 network.sent‚720124 disk.dm-0.in‚18.09375 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.40727874920263% disk./.usageGB‚1258.9536514282227 disk.rbd1.out‚ 4.138671875 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0$ disk.nvme0n1.out‚1977.2587890625# proc.memory.rssMB‚ 2869.77734375 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.sda.in‚0.015625$ memory_percent‚61.20832761128744 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 26.6953125 disk.drbd1965.out‚0 disk.nvme0n1.in‚18.09375 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1931.in‚0 cpu‚25.058748679177356 disk.drbd1673.out‚0! disk.dm-0.out‚1977.2587890625 disk.drbd1285.out‚0 proc.cpu.threads‚17 network.recv‚102847* proc.memory.percent‚11.677153905232746 disk.sda2.in‚0.015625 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0Æcž‚(:“ ×ÿ‰ÓÙõij gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.411' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.47033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A95ž‚(:ª  ×ÿ‰Óõ´â³% disk./.usageGB‚1258.9536323547363 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚18.09375 disk.drbd2111.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.sda2.out‚ 26.9921875 disk.drbd1931.in‚0 disk.drbd1226.in‚0% memory_percent‚61.207358042399086 network.recv‚106134 disk.rbd1.out‚ 4.2646484375 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.40727768251342 disk.sda.in‚0.015625 disk.drbd2078.out‚0 disk.sda.out‚ 26.9921875 disk.drbd1673.in‚0 network.sent‚797820 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2869.77734375 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0! disk.dm-0.out‚1977.6220703125 disk.drbd1285.out‚0 disk.nvme0n1.in‚18.09375$ disk.nvme0n1.out‚1977.6220703125 disk.drbd1817.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.677153905232746 cpu‚25.061526306536916 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2041.out‚0( proc.memory.availableMB‚ 9533.6796875y0·¾Ÿ‚(:’ æÿ‰ÓÔ‹°³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚232.29' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.42999999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PðFœ0 ‚(:¥  æÿ‰ÓŒØѳ disk.drbd1931.in‚0" disk.dm-0.out‚1977.94677734375 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.sda.out‚27.34375$ memory_percent‚61.20737393697103 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 proc.memory.rssMB‚ 2869.78125 cpu‚25.064986067753786% disk./.usageGB‚1258.9536323547363 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.40727768251342 disk.drbd2118.in‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 9533.67578125 disk.drbd2041.out‚0 disk.sda1.out‚0 network.recv‚106242 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚27.34375 disk.drbd1584.in‚0 disk.drbd1289.out‚0 network.sent‚797928 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.015625 disk.sda2.in‚0.015625 disk.dm-0.in‚19.84375 disk.rbd1.out‚ 4.3388671875 disk.drbd1285.out‚0% disk.nvme0n1.out‚1977.94677734375 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.677169799804688 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚19.84375 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0¤ÛQr3¡‚(:¨  õÿ‰Ó”™Õ³' proc.memory.availableMB‚ 9533.671875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0$ memory_percent‚61.20738983154297* proc.memory.percent‚11.677169799804688 disk.drbd1817.out‚0 disk.dm-0.in‚19.84375 disk.drbd1356.in‚0 disk.nvme0n1.in‚19.84375 disk.drbd1550.out‚0 network.sent‚860236 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 proc.memory.rssMB‚ 2869.78125 cpu‚25.06807585893417 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 27.72265625 disk.sda2.out‚ 27.72265625 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚1978.11865234375 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 4.7490234375 disk.drbd1552.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.40727768251342 disk.drbd1965.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1938.in‚0 network.recv‚108388% disk./.usageGB‚1258.9536323547363 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1931.in‚0" disk.dm-0.out‚1978.11865234375 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0N¥“럢‚(:” õÿ‰ÓÐι³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.271' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.42366666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<µGÇž£‚(:“ „€ŠÓÖÖó gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.815' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.93833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ó¥û,¤‚(:¡  „€ŠÓßÞâ³ disk.drbd2041.in‚0" disk.nvme0n1.out‚ 1978.43359375 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.677169799804688 disk.sda.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 proc.memory.rssMB‚ 2869.78125 cpu‚25.071068220766662 disk.drbd1938.in‚0 disk.dm-0.in‚19.84375) disk./.usagePercent‚70.40727874920263 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0$ memory_percent‚61.20738983154297 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚19.84375 disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.drbd2041.out‚0 network.sent‚860302 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 4.7490234375 network.recv‚108454 disk.drbd2078.out‚0% disk./.usageGB‚1258.9536514282227 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.dm-0.out‚ 1978.43359375 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0' proc.memory.availableMB‚ 9533.671875 disk.drbd1673.in‚0 disk.sda2.out‚ 28.4296875 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.sda.out‚ 28.4296875„D“¥‚(:’ “€ŠÓ˜ˆë³ gpu.0.gpu‚14.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚196.324' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.44133333333335 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Ù"¦‚(:—  “€ŠÓ͹‹´ proc.memory.rssMB‚ 2869.78125 network.sent‚860410 disk.drbd1584.in‚0 disk.drbd1938.out‚0# disk./.usageGB‚1258.95365524292 disk.nvme0n1.in‚19.84375 disk.sda.in‚0.015625 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0) proc.memory.availableMB‚ 9533.50390625 cpu‚25.07470405883863 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.out‚ 1978.609375 disk.drbd1965.in‚0 disk.drbd1356.in‚0$ memory_percent‚61.20807329813639 disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 28.81640625 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17 network.recv‚108562 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1226.in‚0 disk.sda.out‚ 28.81640625 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.dm-0.in‚19.84375 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.40727896254046 disk.rbd1.out‚4.875 disk.drbd1285.in‚0 disk.drbd1289.out‚0 disk.dm-0.out‚ 1978.609375* proc.memory.percent‚11.677169799804688 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1510.out‚0 disk.drbd1356.out‚0ið7[0§+ _wandbruntime‚983  _runtime‚983l}&Ũ¿ ' lr-AdamW/pg1‚0.00011701796159952987 ' lr-AdamW/pg2‚0.00011701796159952987 trainer/global_step‚549 $ _timestamp‚1.784840220180759e+09 _runtime‚ 983.538037602 _step‚412¥ïÉ©Ã _runtime‚ 983.538037602 _step‚41 ' lr-AdamW/pg1‚0.00011701796159952987 ' lr-AdamW/pg2‚0.00011701796159952987 trainer/global_step‚549 $ _timestamp‚1.784840220180759e+09)s‡ñ±0ª+ _wandbruntime‚983  _runtime‚983¤95JÖ'«ÐO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5059424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1661210 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 568539709577 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10062 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345435136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1661566 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43462695913472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156555309587609 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 568147309056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611278 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155986769878032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565747 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3125166 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 697133972480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55462400 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 697078510080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9765182 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4695690 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1463600 % _timestamp‚1.7848402201818974e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156616991771136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156616991771136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709851648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12297 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4695690 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44159774423552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156618363379712 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156618363379712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10068 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 569492744192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155986744450064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚ 983.539486047 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156048870635520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10068 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10062 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345435136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1661566 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43462695913472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156555309587609 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 568147309056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12222 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611278 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batcGÃ#¾² h_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155986769878032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565747 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332042 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3125166 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 697133972480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55462400 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 697078510080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9765182 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4695690 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1463600 % _timestamp‚1.7848402201818974e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156616991771136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156616991771136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709851648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12297 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4695690 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44159774423552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039246 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371608576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156618363379712 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156618363379712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10068 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 569492744192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155986744450064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚ 983.539486047 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156048870635520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3130347 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1466240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43541282279424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53358080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 570515259392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4704063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4704063 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9772563 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1664459 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 569169817088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156898295185408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 698327793152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156327779926016 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5078582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156327779926016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚ 985.018223889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44239610072576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156327753752576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156265571651808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5068506 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 570515259392 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3130699 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 698381151232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 $ _timestamp‚1.784840221660584e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4704057 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 569169817088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 _step‚43 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9782645 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9782645 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44240031521792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5078582 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156265546223840K »ã&¯ÝM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371615744 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9772563 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1664459 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 569169817088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156898295185408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611390 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680171520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709851648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371615744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 698327793152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10082 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156327779926016 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5078582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156327779926016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12297 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚ 985.018223889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44239610072576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156327753752576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156265571651808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5068506 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 570515259392 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3130699 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 698381151232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565747 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 $ _timestamp‚1.784840221660584e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4704057 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 569169817088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10076 _step‚43 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9782645 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9782645 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709851648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442304 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44240031521792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12297 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5078582 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156265546223840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156327753752576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43541650370560 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3130347 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1466240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43541282279424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53358080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 570515259392 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4704063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4704063+ Á³Ë0°+ _wandbruntime‚985  _runtime‚985ž>i±ú /loss/FRt1/assignment_loss‚4.115683189402442 2loss/SRqqt1/assignment_loss‚1.7688485581412094 2loss/SRqqt2/assignment_loss‚1.4392362769832174 _step‚44 0loss/FBt1/assignment_loss‚0.2240548508056918 0loss/SRqqt1/detection_loss‚3.472826553149693 1loss/SRqqt2/detection_loss‚0.8326317882711487 .loss/FBt1/detection_loss‚4.326617056320382 % _timestamp‚1.7848402216945076e+09 /loss/FRt2/assignment_loss‚3.467037745490061 1loss/FBt2/assignment_loss‚0.16557715232757214 0loss/FRt1/detection_loss‚0.43271662182025117 /loss/FRt2/detection_loss‚1.3726928293086471 &loss/total_loss‚28.890426966352937 .loss/FBt2/detection_loss‚7.272504344332621 epoch‚1 trainer/global_step‚549 _runtime‚ 985.051771388©ýß²þ /loss/FRt2/assignment_loss‚3.467037745490061 1loss/FBt2/assignment_loss‚0.16557715232757214 0loss/FRt1/detection_loss‚0.43271662182025117 /loss/FRt2/detection_loss‚1.3726928293086471 &loss/total_loss‚28.890426966352937 .loss/FBt2/detection_loss‚7.272504344332621 epoch‚1 trainer/global_step‚549 _runtime‚ 985.051771388 /loss/FRt1/assignment_loss‚4.115683189402442 2loss/SRqqt1/assignment_loss‚1.7688485581412094 2loss/SRqqt2/assignment_loss‚1.4392362769832174 _step‚44 0loss/FBt1/assignment_loss‚0.2240548508056918 0loss/SRqqt1/detection_loss‚3.472826553149693 1loss/SRqqt2/detection_loss‚0.8326317882711487 .loss/FBt1/detection_loss‚4.326617056320382 % _timestamp‚1.7848402216945076e+09,c*žÒž³‚(:“ ¢€ŠÓºµÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚277.148' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.38266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀñG+´‚(:   ¢€ŠÓÜÅò³ disk.sda2.in‚0.015625 disk.drbd1584.out‚0 disk.drbd1673.out‚0$ disk.nvme0n1.out‚1979.0146484375 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1356.out‚0 cpu‚25.077771344145553 disk.drbd1931.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 29.1015625 disk.drbd1965.out‚0 disk.drbd2111.out‚0( proc.memory.availableMB‚ 9533.2109375 network.sent‚860476) disk./.usagePercent‚70.40727896254046 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1226.out‚0# disk./.usageGB‚1258.95365524292 disk.drbd1965.in‚0* proc.memory.percent‚11.677169799804688 disk.dm-0.in‚19.84375 disk.drbd2078.in‚0 disk.drbd1931.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 proc.memory.rssMB‚ 2869.78125 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 network.recv‚108628 disk.drbd1584.in‚0 disk.sda.out‚ 29.1015625! disk.dm-0.out‚1979.0146484375 disk.rbd1.out‚ 4.9609375 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0# memory_percent‚61.2092653910319 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚19.84375 disk.sda1.in‚0 disk.drbd1828.out‚0¨^s±µ‚(:ª  ±€ŠÓôÑ„´# memory_percent‚61.2267812093099 disk.drbd1931.in‚0 disk.rbd1.out‚ 5.4921875 disk.nvme0n1.in‚ 25.23046875 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.015625 disk.drbd1289.out‚0 disk.dm-0.in‚ 25.23046875 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 cpu‚25.086750566029913 proc.cpu.threads‚17 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2873.7421875 network.recv‚109757 disk.drbd1673.in‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.sda1.out‚0) disk./.usagePercent‚70.40727789:Xš585127 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.693286895751953 disk.sda.out‚ 29.69140625 disk.sda2.out‚ 29.69140625! disk.dm-0.out‚1979.1904296875 disk.drbd1584.out‚0 disk.drbd1965.in‚0$ disk.nvme0n1.out‚1979.1904296875& proc.memory.availableMB‚ 9528.90625 network.sent‚866020 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0% disk./.usageGB‚1258.9536361694336 disk.drbd2111.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0Šßv9Ÿ¶‚(:” ±€ŠÓŸ¨»µ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚303.254' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.08466666666666 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ5Ð ž·‚(:“ À€ŠÓ»¥Ò³ gpu.0.gpu‚95.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚284.468' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.82266666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•>Ø[7¸‚(:¬  À€ŠÓÈð³ disk.drbd1550.out‚0 disk.sda1.out‚0 disk.sda.out‚ 29.93359375 disk.sda2.in‚0.015625 disk.drbd1938.in‚0" disk.dm-0.out‚1979.56494140625) disk./.usagePercent‚70.40727789585127% disk./.usageGB‚1258.9536361694336 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 network.sent‚866086 disk.drbd1584.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚1979.56494140625 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd1828.in‚0% memory_percent‚61.245107650756836 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0) proc.memory.percent‚11.70941988627116 network.recv‚109823 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 30.859375 disk.drbd2041.in‚0 disk.sda.in‚0.015625 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 30.859375 cpu‚25.12301295107954 disk.drbd1817.out‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 5.4921875 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2877.70703125 disk.sda2.out‚ 29.93359375 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 9524.40234375 Ìt2¹- _wandbruntime‚1021  _runtime‚1021G()r£;ºw =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.24086619913578033 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9649764895439148 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2259230762720108 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.20985107123851776 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.016480453312397003 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.21253682672977448 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9877551198005676 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975722432136536 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974437952041626 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.001400560257025063 7validation_average_jet_accuracy‚0.15773048996925354 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.32584840059280396 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34090909361839294 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.019117670133709908 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.013390542939305305 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0022183770779520273 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008710800902917981 % _timestamp‚1.7848402584024205e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9915726780891418 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2811509668827057 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09578455239534378 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0957116112112999 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2438676804304123 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21206693351268768 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1115344986319542 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1785714328289032 ) Purity/1FRt1SRqqt2FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.16754400730133057 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9697192311286926 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9618012309074402 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.012892454862594604 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07279235869646072 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07279235869646072 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.11483517289161682 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚28.848892286876907 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997778594493866 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4871237874031067 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.26353010535240173 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.052910469472408295 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.04977027699351311 +particle/specificity‚0.8764328956604004 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.05098217353224754 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.04689035192131996 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.14276151359081268 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.35602694749832153 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_2‚0.43878647685050964 (particle/accuracy‚0.8028215169906616 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5628286600112915 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3524688184261322 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14145682752132416 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14255622029304504 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12632820010185242 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.25028499960899353 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976190328598022 )jet/accuracy_1_of_3‚0.576144814491272 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9974151253700256 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.25028499960899353 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.24980881810188293 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.01515151560306549 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.08492221683263779 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09020449966192245 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7651408314704895 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0034200656227767467 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.10353415459394455 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08602558821439743 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14167429506778717 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.24567784368991852 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9293256998062134 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 +particle/sensitivity‚0.5334069728851318 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.08874756097793579 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9270833134651184 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.19213047623634338 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ,jet/accuracy_3_of_3‚0.005788149777799845 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 /particle/accuracy_3_of_4‚0.9965986609458923 'particle/f_score‚0.5372231006622314 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2132430523633957 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0008210180676542222 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.20648670196533203 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9928751587867737 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.25597915053367615 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2698531448841095 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2549075782299042 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.21053309738636017 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.133137509226799 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13494420051574707 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9869980216026306 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2717994153499603 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.39393937587738037 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6046554446220398 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0881887748837471 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08802960067987442 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.17975768446922302 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9837867617607117 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.2077767550945282 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_4‚0 particle/accuracy_1_of_3‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.25027692317962646 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.990122377872467 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.042796093970537186 +jet/accuracy_2_of_4‚0.11074099689722061 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05066189914941788 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.04646916314959526 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2852879464626312 )jet/accuracy_1_of_4‚0.583104133605957 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.012910531833767891 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9888315796852112 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06542596966028214 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.05004409700632095 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4125217795372009 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9978991746902466 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.013390542939305305 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.18205127120018005 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1939925253391266 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.21053309738636017 particle/accuracy_1_of_1‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.1785714328289032 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13116398453712463 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 trainer/global_step‚559 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.19728276133537292 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 +jet/accuracy_2_of_3‚0.14094902575016022 particle/accuracy_2_of_3‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6552535891532898 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2454620599746704 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 epoch‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.00822467915713787 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09235209226608276 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.023806288838386536 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.013924176804721355 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09334415942430496 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.010286406613886356 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.29026028513908386 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9984126687049866 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.26435038447380066 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014880952658131719 /particle/accuracy_3_of_3‚0.9991883039474487 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09819702059030533 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.20007577538490295 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8430272936820984 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 _runtime‚ 1021.76052912 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9967532157897949 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7667803168296814 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9842729568481445 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997778594493866 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.39393937587738037 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9876165390014648 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1796272099018097 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02083333395421505 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.40608352422714233 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895468950271606 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.20942188799381256 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9734811782836914 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0037649262230843306 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 +jet/accuracy_1_of_1‚0.12463054805994034 particle/accuracy_2_of_2‚1 _step‚45 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.29529789090156555 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20483502745628357 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936787486076355 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5001410841941833 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02083333395421505 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9299529194831848 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.00804086122661829 ,jet/accuracy_3_of_4‚0.003759398590773344 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.019547780975699425 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.013466433621942997 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.003486842382699251 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3205128014087677 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844ÎL§j7»¡w 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.019547780975699425 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.013466433621942997 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.003486842382699251 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3205128014087677 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.24086619913578033 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9649764895439148 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2259230762720108 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.20985107123851776 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.016480453312397003 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.21253682672977448 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9877551198005676 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975722432136536 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974437952041626 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.001400560257025063 7validation_average_jet_accuracy‚0.15773048996925354 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.32584840059280396 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34090909361839294 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.019117670133709908 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.013390542939305305 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0022183770779520273 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008710800902917981 % _timestamp‚1.7848402584024205e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9915726780891418 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2811509668827057 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09578455239534378 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0957116112112999 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2438676804304123 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21206693351268768 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1115344986319542 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1785714328289032 ) Purity/1FRt1SRqqt2FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.16754400730133057 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9697192311286926 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9618012309074402 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.012892454862594604 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07279235869646072 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07279235869646072 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.11483517289161682 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚28.848892286876907 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997778594493866 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4871237874031067 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.26353010535240173 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.052910469472408295 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.04977027699351311 +particle/specificity‚0.8764328956604004 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.05098217353224754 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.04689035192131996 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.14276151359081268 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.35602694749832153 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_2‚0.43878647685050964 (particle/accuracy‚0.8028215169906616 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5628286600112915 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3524688184261322 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14145682752132416 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14255622029304504 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12632820010185242 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.25028499960899353 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976190328598022 )jet/accuracy_1_of_3‚0.576144814491272 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9974151253700256 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.25028499960899353 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.24980881810188293 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.01515151560306549 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.08492221683263779 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09020449966192245 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7651408314704895 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0034200656227767467 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.10353415459394455 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08602558821439743 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14167429506778717 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.24567784368991852 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9293256998062134 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 +particle/sensitivity‚0.5334069728851318 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.08874756097793579 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9270833134651184 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.19213047623634338 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ,jet/accuracy_3_of_3‚0.005788149777799845 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 /particle/accuracy_3_of_4‚0.9965986609458923 'particle/f_score‚0.5372231006622314 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2132430523633957 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0008210180676542222 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.20648670196533203 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9928751587867737 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.25597915053367615 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2698531448841095 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2549075782299042 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.21053309738636017 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.133137509226799 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13494420051574707 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9869980216026306 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2717994153499603 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.39393937587738037 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6046554446220398 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0881887748837471 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08802960067987442 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.17975768446922302 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9837867617607117 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.2077767550945282 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_4‚0 particle/accuracy_1_of_3‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.25027692317962646 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.990122377872467 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.042796093970537186 +jet/accuracy_2_of_4‚0.11074099689722061 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05066189914941788 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.04646916314959526 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2852879464626312 )jet/accuracy_1_of_4‚0.583104133605957 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.012910531833767891 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9888315796852112 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06542596966028214 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.05004409700632095 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4125217795372009 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9978991746902466 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.013390542939305305 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.18205127120018005 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1939925253391266 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.21053309738636017 particle/accuracy_1_of_1‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.1785714328289032 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13116398453712463 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 trainer/global_step‚559 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.19728276133537292 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 +jet/accuracy_2_of_3‚0.14094902575016022 particle/accuracy_2_of_3‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6552535891532898 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2454620599746704 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 epoch‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.00822467915713787 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09235209226608276 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.023806288838386536 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.013924176804721355 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09334415942430496 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.010286406613886356 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.29026028513908386 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9984126687049866 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.26435038447380066 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014880952658131719 /particle/accuracy_3_of_3‚0.9991883039474487 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09819702059030533 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.20007577538490295 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8430272936820984 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 _runtime‚ 1021.76052912 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9967532157897949 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7667803168296814 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9842729568481445 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997778594493866 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.39393937587738037 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9876165390014648 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1796272099018097 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02083333395421505 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.40608352422714233 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895468950271606 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.20942188799381256 º=7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9734811782836914 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0037649262230843306 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 +jet/accuracy_1_of_1‚0.12463054805994034 particle/accuracy_2_of_2‚1 _step‚45 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.29529789090156555 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20483502745628357 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936787486076355 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5001410841941833 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02083333395421505 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9299529194831848 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.00804086122661829 ,jet/accuracy_3_of_4‚0.003759398590773344-U¡ šž¼‚(:“ Ï€ŠÓàŠÝ³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚270.727' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.24233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DFLÖ5½‚(:ª  Ï€ŠÓôÔý³ disk.sda.out‚ 30.26953125 disk.drbd1673.in‚0 disk.drbd1965.in‚0 network.sent‚866194 disk.drbd1552.out‚0 disk.sda2.in‚0.015625 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2877.55859375# disk./.usageGB‚1259.89994430542 disk.sda.in‚0.015625 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.rbd1.out‚ 5.6640625 disk.drbd1285.in‚0% disk.nvme0n1.out‚2950.04931640625 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 30.859375 disk.drbd1965.out‚0$ memory_percent‚65.19856452941895 cpu‚25.12011140599725 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0" disk.dm-0.out‚2950.04931640625 disk.drbd2041.in‚0* proc.memory.percent‚11.708815892537434 disk.sda2.out‚ 30.26953125 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2118.in‚0) proc.memory.availableMB‚ 8552.80078125 disk.nvme0n1.in‚ 30.859375 network.recv‚109931) disk./.usagePercent‚70.46020040069284 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0½bÂ+¾‚(:   Þ€ŠÓߊé³ disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 5.6640625 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.dm-0.out‚ 2950.7265625 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 30.62890625 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.sda2.in‚0.015625 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0# proc.memory.rssMB‚ 2878.66015625 cpu‚25.122148037638762! disk.nvme0n1.out‚ 2950.7265625 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 30.62890625 disk.dm-0.in‚ 30.859375 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0# disk./.usageGB‚1259.89994430542 disk.nvme0n1.in‚ 30.859375 network.recv‚112286* proc.memory.percent‚11.713298161824543 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0$ memory_percent‚65.20398457845053 network.sent‚933460 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0& proc.memory.availableMB‚ 8551.46875) disk./.usagePercent‚70.46020040069284 disk.drbd2078.out‚03ðË’¿‚(:‡ Þ€ŠÓ¬ŒÅ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚329.949' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚109.983 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u ÛÞžÀ‚(:“ 퀊Óªçѳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚278.071' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.69033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r³§/Á‚(:¤  퀊Ó£¡ó³ disk.drbd1356.in‚0 disk.rbd1.out‚ 5.7265625 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 31.08203125 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 31.08203125# proc.memory.rssMB‚ 2878.66015625 network.recv‚112352 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 30.859375 disk.sda2.in‚0.015625 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0& proc.memory.availableMB‚ 8551.59375$ memory_percent‚65.20347595214844) disk./.usagePercent‚70.45999644971698 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.713298161824543 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 cpu‚25.12389034333046$ disk./.usageGB‚1259.896297454834 disk.drbd1285.out‚0 disk.drbd1226.in‚0 network.sent‚933526 disk.drbd1931.in‚0 disk.dm-0.in‚ 30.859375 disk.drbd1356.out‚0 disk.drbd2041.in‚0# disk.nvme0n1.out‚3920.646484375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.drbd1550.in‚0 disk.dm-0.out‚3920.646484375 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0„Gñ7‚(:¬  ü€ŠÓÊÞ³ disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1510.out‚0 disk.drbd1510.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0" disk.dm-0.out‚3921.44677734375 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 31.28515625 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 31.28515625 disk.drbd1931.in‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 8551.57421875 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 30.859375 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 cpu‚25.12604495467785 disk.sda.in‚0.015625* proc.memory.percent‚11.713298161824543 proc.cpu.threads‚17 network.sent‚992719 disk.drbd1817.out‚0 network.recv‚114645) disk./.usagePercent‚70.45994610198655 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚3921.44677734375% disk./.usageGB‚1259.8953971862793 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd1931.out‚0 disk.dm-0.in‚ 30.859375 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.7265625 disk.sda1.out‚0 disk.drbd2111.in‚0$ memory_percent‚65.20355542500815'óžÃ‚(:“ ü€ŠÓ»àº³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚269.245' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.74833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´W‘2Ä- _wandbruntime‚1094  _runtime‚1094|1À»ÆÅÀ ' lr-AdamW/pg1‚0.00011697285782091943 ' lr-AdamW/pg2‚0.00011697285782091943 trainer/global_step‚599 % _timestamp‚1.7848403315680075e+09 _runtime‚ 1094.92525553 _step‚46¨*PÎÊÆÄ % _timestamp‚1.7848403315680075e+09 _runtime‚ 1094.92525553 _step‚46 ' lr-AdamW/pg1‚0.00011697285782091943 ' lr-AdamW/pg2‚0.00011697285782091943 trainer/global_step‚599.À æÛ2Ç- _wandbruntime‚1094  _runtime‚1094ÊDEÛ'ÈÕO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172555011885056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173181312622592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3438408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48208003003392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10063 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5562683 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47441381815296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172485834290032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _step‚47 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5572746 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1835916 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5203472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173179941013504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 623907820413 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 626300737536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172554985711616 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 626300737536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423553024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173109742110445 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10766155 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 624955301888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852160 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3438062 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173111112149695 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 trainer/global_step‚599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173179941013504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12223 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48208426556416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚172555011885056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 767044741120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10776224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1602487 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5562683 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47441013724160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 624955301888 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5203478 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚1094.926579548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 625252431695 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5203472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172554985711616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1602492 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5203478 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5562683 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47441013724160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 624955301888 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5203478 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚1094.926579548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 625252431695 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5203472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172554985711616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1602492 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5203478 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173179941013504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 623907820413 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 626300737536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172554985711616 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 626300737536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423553024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173109742110445 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10766155 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 624955301888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852160 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3438062 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173111112149695 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 trainer/global_step‚599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346/aÝvžÊ‚(:“ ‹ŠÓЀ׳ gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚214.709' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.56966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jéß6Ë‚(:«  ‹ŠÓó§ø³ disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 network.sent‚998366 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚3921.78271484375$ memory_percent‚65.20560582478842 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0( proc.memory.availableMB‚ 8551.0703125 disk.drbd1938.out‚0" disk.dm-0.out‚3921.78271484375 disk.drbd1817.in‚0 disk.sda.in‚0.015625 disk.drbd2078.out‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.713298161824543 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.sda.out‚ 31.578125 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.sda1.out‚0 disk.dm-0.in‚ 30.859375 disk.rbd1.out‚ 5.8525390625 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2878.66015625 network.recv‚115659 disk.nvme0n1.in‚ 30.859375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0) disk./.usagePercent‚70.45995911559484% disk./.usageGB‚1259.8956298828125 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 31.578125 disk.drbd2111.out‚0 cpu‚25.127904508327635 -X2Ì- _wandbruntime‚1096  _runtime‚1096>;_XÍßM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3443963 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚335 4.DeviceStatsMonitor.on_train_batch_e‘éý$nd/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3443628 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48288266823680 _runtime‚1096.402118446 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5211845 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1838501 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10793687 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172764661491776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173389564685861 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173459872812032 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768298642432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10083 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 625977809920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12223 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12298 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5581842 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1605132 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚330 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 _step‚48 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5581842 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5211839 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 626273233447 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10793687 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173461244428288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172833921175552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 627323252736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5211839 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 625977809920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5571765 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172833921175552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173390934725223^/ˆ,é&ÎãM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1838501 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10793687 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172764661491776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173389564685861 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173459872812032 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768298642432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10083 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 625977809920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12223 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12298 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5581842 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1605132 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚330 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 _step‚48 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5581842 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5211839 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 626273233447 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10793687 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173461244428288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172833921175552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 627323252736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5211839 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 625977809920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5571765 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172833921175552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173390934725223 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3443963 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚335 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3443628 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48288266823680 _runtime‚1096.402118446 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5211845 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚560~ZÁ^2Ï- _wandbruntime‚1096  _runtime‚1096`fî‰Ðý /loss/FRt1/assignment_loss‚4.063634086796578 2loss/SRqqt2/assignment_loss‚1.5141551251293521 1loss/FBt2/assignment_loss‚0.13229240070475845 /loss/FRt1/detection_loss‚0.4553817376281586 /loss/FRt2/detection_loss‚1.3358378521779213 /loss/FBt1/detection_loss‚4.3156618773937225 /loss/FBt2/detection_loss‚2.1227026577656565 &loss/total_loss‚23.880191231612454 epoch‚2 trainer/global_step‚599 _runtime‚1096.438827985 /loss/FRt2/assignment_loss‚3.312479230520785 2loss/SRqqt1/assignment_loss‚1.8315419353833744 0loss/FBt1/assignment_loss‚0.3581034325953494 1loss/SRqqt1/detection_loss‚3.6073442840652636 1loss/SRqqt2/detection_loss‚0.8310566114515325 % _timestamp‚1.7848403330816386e+09 _step‚49 Ñ 1loss/SRqqt1/detection_loss‚3.6073442840652636 1loss/SRqqt2/detection_loss‚0.8310566114515325 % _timestamp‚1.7848403330816386e+09 _step‚49 /loss/FRt1/assignment_loss‚4.063634086796578 2loss/SRqqt2/assignment_loss‚1.5141551251293521 1loss/FBt2/assignment_loss‚0.13229240070475845 /loss/FRt1/detection_loss‚0.4553817376281586 /loss/FRt2/detection_loss‚1.3358378521779213 /loss/FBt1/detection_loss‚4.3156618773937225 /loss/FBt2/detection_loss‚2.1227026577656565 &loss/total_loss‚23.880191231612454 epoch‚2 trainer/global_step‚599 _runtime‚1096.438827985 /loss/FRt2/assignment_loss‚3.312479230520785 2loss/SRqqt1/assignment_loss‚1.8315419353833744 0loss/FBt1/assignment_loss‚0.35810343259534941¢”J”Ò‚(:‰ šŠÓ©¢á³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.324' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò3!8Ó‚(:­  šŠÓ‹´ disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 5.8525390625 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2877.765625 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 8552.12109375 disk.drbd1584.out‚0* proc.memory.percent‚11.709658304850262 disk.drbd1289.in‚0" disk.dm-0.out‚3922.12158203125 disk.nvme0n1.in‚ 30.859375 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 31.87109375 disk.drbd1673.in‚0 disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚3922.12158203125 network.sent‚998432 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0$ disk./.usageGB‚1259.897663116455 disk.drbd1965.out‚0 network.recv‚115725 disk.drbd1289.out‚0 disk.dm-0.in‚ 30.859375 disk.sda2.out‚ 31.87109375 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 cpu‚25.129503031462882) disk./.usagePercent‚70.46007282466401 disk.drbd1817.in‚0$ memory_percent‚65.20133018493652 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17hG6ížÔ‚(:“ ©ŠÓÀŠÆ³ gpu.0.gpu‚87.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚184.757' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.58566666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v>@?Õ‚(:´  ©ŠÓÈå³ disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0( proc.memory.percent‚11.7096741994222 network.sent‚ 1.001518e+06 disk.drbd1552.out‚0 disk.sda.in‚0.015625 disk.drbd2118.in‚0% disk.nvme0n1.out‚3922.32080078125 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0$ memory_percent‚65.20312627156576 disk.drbd1356.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 network.recv‚116765 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 32.8046875 disk.drbd1226.in‚0% disk./.usageGB‚1259.8976440429688 disk.drbd1817.out‚0 disk.sda.out‚ 32.32421875 disk.drbd1285.in‚0 cpu‚25.131504188219335 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 5.9150390625 disk.drbd1550.out‚0( disk./.usagePercent‚70.4600717579748 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 32.8046875 disk.sda1.in‚0" disk.dm-0.out‚3922.32080078125( proc.memory.availableMB‚ 8551.6796875# proc.memory.rssMB‚ 2877.76953125 disk.sda2.out‚ 32.32421875 disk.drbd2078.in‚0 disk.drbd1226.out‚0sÿVŸÖ‚(:” ¸ŠÓæŒØ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.081' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.69366666666669 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N\Ë7ׂ(:¬  ¸ŠÓšãó³ disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 8551.82421875 disk.drbd1510.in‚0 disk.drbd2078.out‚0$ memory_percent‚65.20253817240396" disk.dm-0.out‚3923.59033203125 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 5.9150390625 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.46007026460993 disk.drbd2118.out‚0 disk.sda2.out‚32.6875 disk.drbd1584.out‚0 disk.dm-0.in‚ 32.8046875( proc.memory.percent‚11.7096741994222 proc.cpu.threads‚17 disk.sda1.out‚0$ disk./.usageGB‚1259.897617340088 disk.nvme0n1.in‚ 32.8046875 disk.drbd1550.out‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0 network.recv‚116831 disk.drbd1552.out‚0 disk.drbd2041.in‚0 cpu‚25.13301366939368 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda.out‚32.6875 disk.drbd1289.in‚0 network.sent‚ 1.001584e+06% disk.nvme0n1.out‚3923.59033203125 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1673.out‚0# proc.memory.rssMB‚ 2877.76953125ZZ‹øžØ‚(:“ ÇŠÓü½¦³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.33366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UnËp>Ù‚(:³  ÇŠÓãŽÈ³ disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.nvme0n1.in‚ 32.8046875 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 32.9453125 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 8550.93359375 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0# memory_percent‚65.2061621348063 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.015625 disk.sda2.in‚0.015625 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 32.9453125 disk.drbd1938.out‚0 disk.dm-0.in‚ 32.8046875# proc.memory.rssMB‚ 2878.66015625 network.recv‚116981 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0% disk.nvme0n1.out‚3923.73095703125* proc.memory.percent‚11.713298161824543 cpu‚25.134715414002546) disk./.usagePercent‚70.46007026460993 disk.drbd1828.in‚0$ disk./.usageGB‚1259.897617340088 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 network.sent‚ 1.001734e+06 disk.drbd1226.in‚0 disk.drbd2118.out‚0" disk.dm-0.out‚3923.73095703125 disk.rbd1.out‚ 6.041015625ýiÖí;Ú‚(:°  ÖŠÓ—Ò€´ disk.dm-0.in‚ 32.8046875 disk.drbd1285.out‚0% disk.nvme0n1.out‚3924.80224609375 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2878.66015625 network.sent‚ 1.0018e+06 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.sda.out‚ 33.4609375 disk.drbd2041.out‚0 cpu‚25.13654834648201 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0) proc.memory.availableMB‚ 8550.93359375$ disk./.usageGB‚1259.897617340088 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.015625 disk.nvme0n1.in‚ 32.8046875 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 6.041015625* proc.memory.percent‚11.713298161824543 proc.cpu.threads‚17 disk.drbd1550.in‚0" disk.dm-0.out‚3924.80224609375 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.46007026460993 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0# memory_percent‚65.2061621348063 disk.sda2.out‚ 33.4609375 disk.drbd1289.in‚0 network.recv‚117047 disk.drbd1965.out‚0 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0ñÛIœÛ‚(:‘ ÖŠÓÞÿß³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.764' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.92133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bLU2Ü- _wandbruntime‚1182  _runtime‚1182€&(çÅÝ¿ ' lr-AdamW/pg2‚0.00011692009804631045 trainer/global_step‚649 $ _timestamp‚1.784840418722412e+09 _runtime‚ 1182.07964232 _step‚50 ' lr-AdamW/pg1‚0.00011692009804631045a&ëÉÞà $ _timestamp‚1.784840418722412e+09 _runtime‚ 1182.07964232 _step‚50 ' lr-AdamW/pg1‚0.00011692009804631045 ' lr-AdamW/pg2‚0.00011692009804631045 trainer/global_step‚6492kÕ2ß- _wandbruntime‚1182  _runtime‚1182S⸵žàÔO trainer/global_step‚649 _step‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 677490293760 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11640457 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚186500476409856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186500450236416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12298 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1981105 SGDeviceStat%)ë<#sMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1980752 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52199859107840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186500450236416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11650526 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5621828 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51370700112896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187100934682053 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10063 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12223 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3715597 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6028698 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186500476409856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6018635 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3715239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1734487 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6028698 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187176595094528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186425922978832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6018635 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52200280563712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51370332021760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 829580450816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 829527086080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 676356314503 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 675011703221 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187177966703616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 677490293760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 $ _timestamp‚1.784840418723447e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10063 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186425948406800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0s­¿tÞ'áØO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186500450236416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11650526 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5621828 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51370700112896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187100934682053 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10063 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12223 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3715597 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6028698 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186500476409856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6018635 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3715239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1734487 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6028698 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187176595094528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186425922978832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6018635 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52200280563712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51370332021760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 829580450816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 829527086080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 676356314503 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 675011703221 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187177966703616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 677490293760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 $ _timestamp‚1.784840418723447e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10063 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186425948406800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚649 _step‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 677490293760 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11640457 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚186500476409856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186500450236416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12298 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1981105 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1980752 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52199859107840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚03ÿˆH2â- _wandbruntime‚1183  _runtime‚11831Eͨå&ãßM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚186779359526912 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1984008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11657906 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11667989 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187456526893056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187380757257469 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 677377116255 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5630189 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 830831253504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187382127296831 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1983654 % _timestamp‚1.7848404202019763e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1737132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425643008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52279692089344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51448918387712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5630189 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6027717 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51449286478848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 _step‚52 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 830773701632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678512808960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5630195 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 676032504861 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 677167366144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6037794 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52280117732352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10083 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚186779359526912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186779385700352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 678512808960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11667989 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12298 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3720781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚354 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚649 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6037794 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187457898509312 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6027717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0랬 äãM MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5630195 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 678512808960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11667989 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12298 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3720781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚354 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚649 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6037794 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187457898509312 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6027717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚186779359526912 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1984008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11657906 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11667989 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187456526893056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187380757257469 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 677377116255 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5630189 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 830831253504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18738212Ý6GU=7296831 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1983654 % _timestamp‚1.7848404202019763e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1737132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425643008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52279692089344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51448918387712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5630189 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6027717 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51449286478848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 _step‚52 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 830773701632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678512808960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5630195 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 676032504861 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 677167366144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6037794 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52280117732352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10083 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚186779359526912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186779385700352 Fí‚(:³  ƒ‚ŠÓ™ßë³ disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0" disk.dm-0.out‚3925.72021484375% disk.nvme0n1.out‚3925.72021484375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.rbd1.out‚ 6.4091796875 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0% disk./.usageGB‚1259.8979110717773 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 34.86328125 disk.drbd1817.out‚0 disk.sda.out‚ 34.86328125 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.sda.in‚0.015625 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.dm-0.in‚ 32.8046875 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.46008669162367 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 32.8046875$ memory_percent‚65.20652770996094 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 cpu‚25.1414447594273 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.713298161824543 disk.sda2.in‚0.015625 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd2111.in‚0& proc.memory.availableMB‚ 8550.84375 network.sent‚ 1.064555e+06 network.recv‚120030Z÷©/”î‚(:‰ ’‚ŠÓÝíó gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚307.401' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.467 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü">ï‚(:³  ’‚ŠÓêâ³& proc.memory.availableMB‚ 8550.84375 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd1817.out‚0$ memory_percent‚65.20652770996094 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 network.sent‚ 1.075389e+06 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.713298161824543" disk.dm-0.out‚3926.18505859375 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 32.8046875 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚3926.18505859375 disk.drbd1938.in‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.46008562493446 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2878.66015625 network.recv‚121500$ disk./.usageGB‚1259.897891998291 disk.dm-0.in‚ 32.8046875 disk.drbd2041.in‚0 disk.rbd1.out‚ 6.4091796875 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0 disk.sda.out‚ 35.35546875 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 35.35546875 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.sda2.in‚0.015625 cpu‚25.14273678440691 proc.cpu.threads‚17¦ùØ¢žð‚(:“ ¡‚ŠÓâÆÖ³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.924' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.97466666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ=ê¿@ñ‚(:µ  ¡‚ŠÓѾð³ disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.015625# proc.memory.rssMB‚ 2877.76953125 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1550.in‚0$ memory_percent‚65.20291964213052 cpu‚25.144381262528537 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 32.8046875 disk.sda1.in‚0 disk.drbd1938.in‚0( disk./.usagePercent‚70.4600869049615 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 35.73828125 disk.sda2.out‚ 35.73828125 disk.drbd1938.out‚0% disk.nvme0n1.out‚3926.36474609375 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 8551.73046875 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 network.recv‚121566 disk.drbd1226.in‚0 disk.drbd1226.out‚0( proc.memory.percent‚11.7096741994222 disk.drbd1673.out‚0 disk.dm-0.in‚ 32.8046875 disk.sda1.out‚0 network.sent‚ 1.075455e+06% disk./.usageGB‚1259.8979148864746 disk.drbd1931.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0" disk.dm-0.out‚3926.36474609375 disk.drbd2111.out‚0 disk.rbd1.out‚ 6.4716796875 disk.sda2.in‚0.015625ÒPѦŸò‚(:” °‚ŠÓԊ׳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.075' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.02499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eå >ó‚(:³  °‚ŠÓÑßõ³ disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 network.sent‚ 1.075521e+06 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 network.recv‚121632% disk./.usageGB‚1259.8979148864746 disk.drbd2111.in‚0$ memory_percent‚65.19767443339029 disk.dm-0.in‚ 32.8046875 disk.drbd2078.in‚0 disk.rbd1.out‚ 6.4716796875 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚3926.63427734375( proc.memory.percent‚11.7096741994222 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2877.76953125 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 8553.01953125" disk.dm-0.out‚3926.63427734375 disk.drbd2111.out‚0 disk.sda.in‚0.015625 disk.sda2.out‚ 36.1171875 disk.sda.out‚ 36.1171875 disk.drbd1931.out‚0 disk.drbd1817.out‚0( disk./.usagePercent‚70.4600869049615 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 32.8046875 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 cpu‚25.145797477672335 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.015625½Êß2ô- _wandbruntime‚1268  _runtime‚1268ªBeÐÇõÁ ' lr-AdamW/pg1‚0.00011685968919130056 ' lr-AdamW/pg2‚0.00011685968919130056 trainer/global_step‚699 % _timestamp‚1.7848405056014228e+09 _runtime‚1268.958666664 _step‚54ìN„ºËöÅ _runtime‚1268.958666664 _step‚54 ' lr-AdamW/pg1‚0.00011685968919130056 ' lr-AdamW/pg2‚0.00011685968919130056 trainer/global_step‚699 % _timestamp‚1.7848405056014228e+096ɽŽ•2÷- _wandbruntime‚1268  _runtime‚1268 L)Ú'øÔO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200445940934656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2125942 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56192284855808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784840505602526e+09 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6484650 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726115586029 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200445914761216 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 727334414336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12223 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 892266445312 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728679849984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200445914761216 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3992783 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56191863399936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12298 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12514759 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3992429 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10063 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6040178 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6040178 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455872 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345435648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201093497292911 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200445940934656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚1268.960078918 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201173249175552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201173249175552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201174620784640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2126291 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 727334414336 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6040172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10063 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201174620784640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200366037095600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200366011667632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728679849984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12524828 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55300018410496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201092127253661 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727460197311 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6484650 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12298 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 892213080576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55299650319360Ñ<ÄùØO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6040178 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6040178 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455872 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345435648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201093497292911 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200445940934656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚1268.960078918 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201173249175552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201173249175552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201174620784640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2126291 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 727334414336 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6040172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10063 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201174620784640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200366037095600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200366011667632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728679849984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12524828 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55300018410496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201092127253661 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727460197311 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6484650 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12298 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 892213080576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55299650319360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200445940934656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2125942 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56192284855808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784840505602526e+09 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6484650 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726115586029 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200445914761216 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 727334414336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12223 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 892266445312 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728679849984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200445914761216 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3992783 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56191863399936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12298 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12514759 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3992429 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10063 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚07¥þ2ú- _wandbruntime‚1270  _runtime‚12704§‘|ä&ûÞM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200644838869376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1270.439164089 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12542291 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200644813441408 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6493746 _step‚56 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12542291 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200724824051712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55378604776448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3998322 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200724850225152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56272121135616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728480999063 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3997967 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1869127 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417254400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55378236685312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201454552590336 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616256 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6048539 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6048539 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727136387669 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201453180974080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200724850225152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2128840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 893516359168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 728356922368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200724824051712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12532208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 728356922368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49163264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729702365184 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6048545 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2129190 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 893467195904 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784840507081526e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6048545 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚201373319868439 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56271703881216 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚201371949829077 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6483669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12532208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1869132)è¢sè&üâM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729702365184 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6048545 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2129190 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 893467195904 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784840507081526e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6048545 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚201373319868439 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56271703881216 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚201371949829077 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6483669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12532208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1869132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200644838869376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1270.439164089 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12542291 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200644813441408 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6493746 _step‚56 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12542291 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200724824051712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55378604776448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3998322 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200724850225152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56272121135616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728480999063 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3997967 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1869127 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417254400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55378236685312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201454552590336 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616256 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6048539 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6048539 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727136387669 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201453180974080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200724850225152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2128840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 893516359168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 728356922368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200724824051712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12532208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 728356922368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49163264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 462885109768ïdiä2ý- _wandbruntime‚1270  _runtime‚1270Á£W3ÿþù %loss/total_loss‚26.47268547020052 /loss/FRt2/detection_loss‚1.3074678730938853 .loss/FBt1/detection_loss‚5.384539486055246 -loss/FBt2/detection_loss‚4.89703282252918 trainer/global_step‚699 _step‚57 0loss/FRt1/assignment_loss‚3.9872234446661814 2loss/SRqqt1/assignment_loss‚1.6667498853057623 _runtime‚1270.473192655 0loss/FBt1/assignment_loss‚0.3529894968305598 1loss/FBt2/assignment_loss‚0.09971157364319065 0loss/SRqqt1/detection_loss‚3.007439850415914 1loss/SRqqt2/detection_loss‚0.7984516322706259 epoch‚2 % _timestamp‚1.7848405071159675e+09 /loss/FRt2/assignment_loss‚3.003735648971414 2loss/SRqqt2/assignment_loss‚1.4238538345882243 /loss/FRt1/detection_loss‚0.5434899218303345uêŠÿý /loss/FRt2/assignment_loss‚3.003735648971414 2loss/SRqqt2/assignment_loss‚1.4238538345882243 /loss/FRt1/detection_loss‚0.5434899218303345 %loss/total_loss‚26.47268547020052 /loss/FRt2/detection_loss‚1.3074678730938853 .loss/FBt1/detection_loss‚5.384539486055246 -loss/FBt2/detection_loss‚4.89703282252918 trainer/global_step‚699 _step‚57 0loss/FRt1/assignment_loss‚3.9872234446661814 2loss/SRqqt1/assignment_loss‚1.6667498853057623 _runtime‚1270.473192655 0loss/FBt1/assignment_loss‚0.3529894968305598 1loss/FBt2/assignment_loss‚0.09971157364319065 0loss/SRqqt1/detection_loss‚3.007439850415914 1loss/SRqqt2/detection_loss‚0.7984516322706259 epoch‚2 % _timestamp‚1.7848405071159675e+099Rúš¦ž€‚(:“ ¿‚ŠÓÕÁ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.233' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.74433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c–@‚(:µ  ¿‚ŠÓåþܳ disk.nvme0n1.in‚ 32.8046875 disk.drbd1510.in‚0 disk.drbd1550.out‚0 network.recv‚121740 disk.drbd1584.out‚0 disk.drbd1356.out‚0 cpu‚25.14719078059612 proc.cpu.threads‚17 disk.drbd1285.out‚0% disk.nvme0n1.out‚3927.15087890625 disk.dm-0.in‚ 32.8046875 disk.drbd1552.in‚0 disk.sda.in‚0.015625 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 6.7138671875 disk.drbd2118.in‚0 disk.sda2.in‚0.015625 disk.drbd1828.out‚0 disk.drbd2111.in‚0 disk.sda2.out‚ 36.54296875 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0% memory_percent‚61.863931020100914( proc.memory.availableMB‚ 9372.3203125% disk./.usageGB‚1259.8984146118164 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2877.76953125( proc.memory.percent‚11.7096741994222 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 network.sent‚ 1.075629e+06 disk.drbd1938.in‚0" disk.dm-0.out‚3927.15087890625 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 36.54296875 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.46011485221865 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0ªËí΂‚(:µ  ΂ŠÓê„ï³ disk.drbd1817.out‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.46011698559707 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 32.8046875 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 cpu‚25.148728002300064 disk.drbd1931.in‚0 disk.rbd1.out‚ 6.7138671875$ memory_percent‚63.00950050354004 proc.cpu.threads‚17 disk.sda.in‚0.015625" disk.dm-0.out‚3927.27978515625 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.sda2.out‚ 36.7578125 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0* proc.memory.pe=©2Hrrcent‚11.713298161824543 disk.drbd2078.in‚0 disk.drbd2041.in‚0$ disk./.usageGB‚1259.898452758789 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd2118.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 network.sent‚ 1.140772e+06 disk.sda1.out‚0) proc.memory.availableMB‚ 9090.78515625 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd1965.in‚0 disk.sda.out‚ 36.7578125 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 32.8046875% disk.nvme0n1.out‚3927.27978515625 network.recv‚124499 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0ªWŒŸƒ‚(:” ΂ŠÓàÃͳ gpu.0.gpu‚74.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚316.484' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚105.49466666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kÁ¯ ž„‚(:“ Ý‚ŠÓùîѳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.972' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.65733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OÈoé@…‚(:µ  Ý‚ŠÓ•Âì³ network.sent‚ 1.14088e+06 network.recv‚124607 disk.drbd1584.out‚0 disk.rbd1.out‚ 6.7763671875# proc.memory.rssMB‚ 2878.66015625% disk.nvme0n1.out‚3927.70458984375 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 32.8046875 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.015625 disk.drbd1965.out‚0$ disk./.usageGB‚1259.898536682129 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 cpu‚25.149863581730056 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 37.13671875 disk.dm-0.in‚ 32.8046875 disk.drbd2111.in‚0 disk.drbd1828.out‚0$ memory_percent‚64.29802576700845) disk./.usagePercent‚70.46012167902957 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0" disk.dm-0.out‚3927.70458984375 disk.drbd1965.in‚0 disk.sda2.out‚ 37.13671875 disk.drbd1938.in‚0( proc.memory.availableMB‚ 8774.1171875 disk.sda.in‚0.015625 disk.rbd1.in‚0* proc.memory.percent‚11.713298161824543 disk.drbd2111.out‚0óñå_ž†‚(:“ ì‚ŠÓˆËÁ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.874' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.29133333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ?¸D>‡‚(:³  ì‚ŠÓ­Üâ³ disk.drbd2041.out‚0$ disk./.usageGB‚1259.898536682129 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 32.8046875 disk.drbd1356.in‚0 network.recv‚124673 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0" disk.dm-0.out‚3927.88818359375 disk.rbd1.out‚ 6.7763671875) disk./.usagePercent‚70.46012167902957 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 32.8046875 cpu‚25.151345538502337 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0$ memory_percent‚65.65729777018228 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625 network.sent‚ 1.140946e+06 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 37.43359375 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.713298161824543 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 37.43359375 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚3927.88818359375% proc.memory.availableMB‚ 8440.0625 disk.drbd1552.in‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0®Š$*>ˆ‚(:³  û‚ŠÓ©³å³ disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.713298161824543 disk.sda2.in‚0.015625 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 32.8046875 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0$ memory_percent‚66.94753964742026 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 cpu‚25.152243375237713) disk./.usagePercent‚70.46012167902957 disk.drbd1584.in‚0 disk.sda2.out‚ 37.6484375 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 32.8046875$ disk.nvme0n1.out‚3928.2568359375 network.recv‚127742 disk.sda.out‚ 37.6484375 disk.drbd1673.in‚0 network.sent‚ 1.208753e+06$ disk./.usageGB‚1259.898536682129 disk.sda1.out‚0 disk.rbd1.out‚ 6.9326171875 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1938.in‚0) proc.memory.availableMB‚ 8122.97265625 disk.drbd1817.out‚0 disk.drbd1673.out‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625! disk.dm-0.out‚3928.2568359375 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17›úž‰‚(:“ û‚ŠÓˆòȳ gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.592' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.19733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôå€î“Š‚(:ˆ ŠƒŠÓžÛͳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.266' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wð%&@‹‚(:µ  ŠƒŠÓƶô³ disk.drbd1673.in‚0$ memory_percent‚68.21907361348471) proc.memory.availableMB‚ 7810.48046875) disk./.usagePercent‚70.46001330340641 disk.drbd1289.in‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.713298161824543 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 32.8046875 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 37.94140625 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 32.8046875 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 cpu‚25.154202082468014 disk.rbd1.out‚ 6.9326171875 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2878.66015625$ disk.nvme0n1.out‚3928.4638671875 disk.drbd1584.out‚0$ disk./.usageGB‚1259.896598815918 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd2118.in‚0 network.sent‚ 1.208861e+06 disk.drbd1226.in‚0 disk.sda2.out‚ 37.94140625 network.recv‚127850 disk.drbd1289.out‚0! disk.dm-0.out‚3928.4638671875 disk.drbd1550.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.015625 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚09ë†2Œ- _wandbruntime‚1356  _runtime‚1356xê`ÆÀ $ _timestamp‚1.784840592665375e+09 _runtime‚1356.022608468 _step‚58 ' lr-AdamW/pg1‚0.00011679163917410664 ' lr-AdamW/pg2‚0.00011679163917410664 trainer/global_step‚749C Û1ÊŽÄ ' lr-AdamW/pg2‚0.00011679163917410664 trainer/global_step‚749 $ _timestamp‚1.784840592665375e+09 _runtime‚1356.022608468 _step‚58 ' lr-AdamW/pg1‚0.00011679163917410664:MœœÌ2- _wandbruntime‚1356  _runtime‚1356DJ',Û'ÕO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215083319825269 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6458528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565751 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 778564080119 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6930539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2272068 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215169903256576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215171274865664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6458528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215169903256576 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60184325973504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215084689864519 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 % _timestamp‚1.7848405926664028e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13399130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1998492 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60183910809088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59229336708096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214391405459456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12298 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2272414 _step‚59 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 _runtime‚1356.023978005 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779869406208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4270906 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214391379286016 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778523970560 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6940602 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59228968616960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214306125784400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777219468837 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13389061 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215171274865664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚214391405459456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214306100356432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6940602 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6930539 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779869406208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚749Q·­ß'‘ÙO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13399130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1998492 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60183910809088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59229336708096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214391405459456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12298 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2272414 _step‚59 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 _runtime‚1356.023978005 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779869406208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4270906 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214391379286016 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778523970560 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6940602 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59228968616960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214306125784400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777219468837 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13389061 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215171274865664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚214391405459456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214306100356432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6940602 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6930539 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779869406208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚749 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215083319825269 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6458528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565751 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 778564080119 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6930539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2272068 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215169903256576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215171274865664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6458528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215169903256576 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60184325973504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215084689864519 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 % _timestamp‚1.7848405926664028e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824;b¿82’- _wandbruntime‚1357  _runtime‚1357=$Îâ( “ÞM trainer/global_step‚749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215451206671360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6939621 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214584927558176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _step‚60 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779546478592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13406510 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4276119 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚214670288576512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6939621 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.smauï.ª¼ll_pool.peak‚ 339100160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6466889 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6949698 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778240270477 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚346 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215451206671360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 956195795456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6466895 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚341 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13416593 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415157248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47066112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2274992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60263750778368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13406510 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13416593 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4276465 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59307554982912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0s‰†‡è&”âM TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779546478592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12298 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2001127 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215363142400685 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚214670314749952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12223 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13416593 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4276465 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59307554982912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215451206671360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6939621 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214584927558176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _step‚60 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779546478592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13406510 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4276119 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10077 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚214670288576512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6939621 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6466889 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345442816 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6949698 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778240270477 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚346 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215451206671360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 956195795456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6466895 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚341 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13416593 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10077 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415157248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47066112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2274992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60263750778368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13406510 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 P›‚(:³  ¨ƒŠÓÅÌû³ disk.drbd1285.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 32.8046875 disk.drbd2078.out‚0 cpu‚25.15640992158251 disk.drbd2041.in‚0 disk.drbd1356.in‚0( proc.memory.availableMB‚ 7579.2734375 network.sent‚ 1.217252e+06 network.recv‚128853 disk.drbd2111.in‚0% disk.nvme0n1.out‚3928.98193359375 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 6.96484375 disk.drbd2118.in‚0) disk./.usagePercent‚70.46001330340641 disk.sda.in‚0.015625 disk.sda2.in‚0.015625 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 38.69921875 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0$ memory_percent‚69.15985743204752$ disk./.usageGB‚1259.896598815918 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚3928.98193359375 disk.nvme0n1.in‚ 32.8046875 disk.drbd1938.in‚0* proc.memory.percent‚11.713298161824543 disk.sda1.in‚0 disk.sda2.out‚ 38.69921875 disk.drbd1931.in‚0 disk.drbd2078.in‚0G9~kžœ‚(:“ ·ƒŠÓí™Þ³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.982' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.32733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S4:‚(:¯  ·ƒŠÓÎ…û³) disk./.usagePercent‚70.46001330340641 disk.dm-0.in‚ 32.8046875 disk.drbd1552.in‚0 cpu‚25.157743007373142 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.sda.out‚ 39.1171875 disk.drbd1938.out‚0 disk.dm-0.out‚3929.544921875 disk.rbd1.out‚ 7.12109375# proc.memory.rssMB‚ 2878.66015625 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.713298161824543 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 39.1171875 disk.sda.in‚0.015625 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 32.8046875 network.sent‚ 1.222877e+06 network.recv‚129804 disk.drbd1289.out‚0 disk.drbd1285.in‚0$ disk./.usageGB‚1259.896598815918 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚3929.544921875 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0$ memory_percent‚69.16003227233887) proc.memory.availableMB‚ 7579.23046875 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0"aŠ™žž‚(:“ ƃŠÓ›Ö³ gpu.0.gpu‚36.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.477' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.15899999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ØD9Ÿ‚(:®  ƃŠÓºõ³ disk.drbd1510.out‚0 network.sent‚ 1.222985e+06 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 7.12109375 disk.dm-0.out‚3930.068359375 disk.drbd1226.in‚0 cpu‚25.15922238363686 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.015625 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 32.8046875$ disk./.usageGB‚1259.896598815918 disk.drbd1965.out‚0# disk.nvme0n1.out‚3930.068359375) proc.memory.availableMB‚ 7579.22265625 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 39.3671875 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.46001330340641 disk.dm-0.in‚ 32.8046875 disk.sda2.in‚0.015625# proc.memory.rssMB‚ 2878.66015625 disk.drbd1673.in‚0 disk.drbd1285.out‚0 network.recv‚129912 disk.drbd2118.in‚0 disk.drbd1550.in‚0$ memory_percent‚69.16006406148276 disk.drbd1356.in‚0* proc.memory.percent‚11.713298161824543 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 39.3671875 disk.drbd2041.out‚0 disk.drbd1285.in‚0 ™¾Ï= ‚(:²  ÕƒŠÓòžÖ³ disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2078.in‚0" disk.dm-0.out‚3930.46533203125 disk.sda.out‚ 39.890625 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0$ memory_percent‚69.16011174519858 disk.drbd1673.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.713298161824543 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.46006941125856 disk.drbd2118.out‚0 disk.sda.in‚0.015625# proc.memory.rssMB‚ 2878.66015625 cpu‚25.16016477145174 network.recv‚132083 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0( proc.memory.availableMB‚ 7579.2109375 disk.drbd1289.in‚0 disk.dm-0.in‚ 32.8046875% disk./.usageGB‚1259.8976020812988 disk.drbd1552.in‚0 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 network.sent‚ 1.285462e+06 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 7.1533203125 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0% disk.nvme0n1.out‚3930.46533203125 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 39.890625 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 32.8046875 disk.drbd1226.in‚0`mBZž¡‚(:“ ÕƒŠÓ¾¶µ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.895' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.29833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ôˆ8¢‚(:­  䃊ÓŸðî³ network.recv‚132191( disk./.usagePercent‚70.4600649311639 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 40.15234375# proc.memory.rssMB‚ 2878.66015625 network.sent‚ 1.28557e+06 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.dm-0.out‚ 3930.8828125) proc.memory.availableMB‚ 7579.20703125 disk.drbd1356.in‚0 disk.sda.in‚0.015625 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 cpu‚25.16141163075716 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0% disk./.usageGB‚1259.8975219726562 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.713298161824543 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 40.15234375 disk.nvme0n1.in‚ 32.8046875 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0.015625 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 7.2470703125 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 32.8046875$ memory_percent‚69.16012763977051 disk.drbd2041.in‚0 disk.drbd1285.in‚0! disk.nvme0n1.out‚ 3930.8828125 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0Öä+fŸ£‚(:” 䃊Ó‰ÜÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚306.914' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.30466666666666 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ;b2¤- _wandbruntime‚1442  _runtime‚1442’bíÆ¥À _step‚62 ' lr-AdamW/pg1‚0.00011671595691452682 ' lr-AdamW/pg2‚0.00011671595691452682 trainer/global_step‚799 $ _timestamp‚1.784840679575298e+09 _runtime‚1442.932543866óýÀwÊ¦Ä ' lr-AdamW/pg1‚0.00011671595691452682 ' lr-AdamW/pg2‚0.00011671595691452682 trainer/global_step‚799 $ _timestamp‚1.784840679575298e+09 _runtime‚1442.932543866 _step‚62>²ú¢(2§- _wandbruntime‚1442  _runtime‚1442! XR¨×O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14273432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10063 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 829713526784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4766Ï›Ž‡P$7565751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64176534027776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1442.935527732 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7396554 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6876872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7396554 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4549139 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229075882436127 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228246189045232 trainer/global_step‚799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2130492 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413067264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228336843810816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 831058962432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229074512396877 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14263363 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1017879022080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚341 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229166557337600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2418647 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7386491 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229166557337600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10063 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2130487 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6876878 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2418311 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228246214473200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12298 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44976128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7386491 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228336869984256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚63 % _timestamp‚1.7848406795764055e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 829713526784 Iµ,ïá'©ÛO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14273432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10063 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172032 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 829713526784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64176534027776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1442.935527732 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7396554 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6876872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7396554 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4549139 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229075882436127 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228246189045232 trainer/global_step‚799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2130492 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413067264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228336843810816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 831058962432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229074512396877 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10069 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14263363 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1017879022080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚341 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229166557337600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12223 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10069 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2418647 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1395809792 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7386491 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229166557337600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10063 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2130487 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6876878 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2418311 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228246214473200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12298 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44976128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7386491 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611282 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228336869984256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚63 % _timestamp‚1.7848406795764055e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345435648 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 829713526784 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚228525016246976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6885245 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14280812 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7405650 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7395573 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12223 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830736034816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2133127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228615753101312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2133132 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 830688764679 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14280812 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 832081477632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616256 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14290895 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229447860752384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228615753101312 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64256371109376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47066112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10083 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6885239 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1019129737728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6885239 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4554345 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1019082671616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228615779274752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 832081477632 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848406810552955e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229446489136128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229354334972293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14290895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 trainer/global_step‚799 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4554685 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚340 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2421218 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚228524990819008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7405650 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7395573ÞÍC“ ¬åM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14290895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12298 trainer/global_step‚799 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4554685 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚340 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2421218 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚228524990819008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7405650 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7395573 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228615779274752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229447860752384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565751 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚228525016246976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6885245 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14280812 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616256 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7405650 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7395573 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12223 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830736034816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2133127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/acµ.ÈèXtive_bytes.large_pool.freed‚228615753101312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2133132 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 830688764679 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14280812 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 832081477632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616256 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14290895 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229447860752384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228615753101312 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64256371109376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47066112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10083 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6885239 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1019129737728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6885239 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4554345 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1019082671616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345442816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228615779274752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172032 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 832081477632 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611394 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848406810552955e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229446489136128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229354334972293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2@+÷Ä!2­- _wandbruntime‚1444  _runtime‚1444õ®ü 0loss/FRt1/assignment_loss‚3.9390231497112573 0loss/FRt2/assignment_loss‚2.5931588920868704 1loss/FBt2/assignment_loss‚0.07641808848518368 trainer/global_step‚799 _runtime‚1444.447121999 _step‚65 0loss/FBt1/assignment_loss‚0.5176977248340864 .loss/FRt2/detection_loss‚1.364036006932357 /loss/FBt1/detection_loss‚10.020935199193415 %loss/total_loss‚27.29481429571184 % _timestamp‚1.7848406810898337e+09 2loss/SRqqt1/assignment_loss‚1.5429168537313684 2loss/SRqqt2/assignment_loss‚1.1679523762416364 1loss/SRqqt1/detection_loss‚2.2450853780539832 1loss/SRqqt2/detection_loss‚0.8154612550278698 /loss/FBt2/detection_loss‚2.3553113141154634 .loss/FRt1/detection_loss‚0.656818057298353 epoch‚2ã)E¯€ % _timestamp‚1.7848406810898337e+09 2loss/SRqqt1/assignment_loss‚1.5429168537313684 2loss/SRqqt2/assignment_loss‚1.1679523762416364 1loss/SRqqt1/detection_loss‚2.2450853780539832 1loss/SRqqt2/detection_loss‚0.8154612550278698 /loss/FBt2/detection_loss‚2.3553113141154634 .loss/FRt1/detection_loss‚0.656818057298353 epoch‚2 0loss/FRt1/assignment_loss‚3.9390231497112573 0loss/FRt2/assignment_loss‚2.5931588920868704 1loss/FBt2/assignment_loss‚0.07641808848518368 trainer/global_step‚799 _runtime‚1444.447121999 _step‚65 0loss/FBt1/assignment_loss‚0.5176977248340864 .loss/FRt2/detection_loss‚1.364036006932357 /loss/FBt1/detection_loss‚10.020935199193415 %loss/total_loss‚27.29481429571184AƒŒ\“°‚(:ˆ óƒŠÓ¨Åȳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚113.478' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©±@±‚(:µ  óƒŠÓ€€è³ disk.drbd1828.out‚0 disk.drbd1510.in‚0% disk./.usageGB‚1259.8975219726562 disk.drbd2118.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0$ memory_percent‚69.16128794352213 disk.drbd1817.in‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 32.8046875 disk.drbd1965.out‚0 disk.drbd2041.out‚0" disk.dm-0.out‚3932.01416015625 disk.drbd2111.out‚0' proc.memory.availableMB‚ 7578.921875 network.sent‚ 1.291157e+06 disk.sda1.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0( disk./.usagePercent‚70.4600649311639 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 network.recv‚133076 disk.sda2.in‚0.015625 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.sda.out‚ 40.51953125 disk.drbd1673.out‚0* proc.memory.percent‚11.713298161824543 cpu‚25.162306107913935 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 7.3955078125 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0% disk.nvme0n1.out‚3932.01416015625 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 32.8046875 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.sda2.out‚ 40.51953125 disk.drbd1289.out‚0 disk.drbd1356.in‚0ø8+<²‚(:±  ‚„ŠÓ˜‰Ò³ disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.sda2.in‚0.015625 network.sent‚ 1.291265e+06 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.sda.out‚ 40.87890625 proc.cpu.threads‚17( disk./.usagePercent‚70.4600649311639 disk.dm-0.out‚3932.529296875 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 40.87890625& proc.memory.availableMB‚ 7578.90625 disk.drbd1931.out‚0% disk./.usageGB‚1259.8975219726562# proc.memory.rssMB‚ 2878.66015625 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 35.02734375 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0# memory_percent‚69.1613515218099 cpu‚25.163353258512696 disk.rbd1.out‚ 7.3955078125 disk.drbd1226.in‚0 disk.drbd2111.out‚0# disk.nvme0n1.out‚3932.529296875 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 35.02734375 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.713298161824543 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 network.recv‚133184hε”³‚(:‰ ‚„ŠÓÉÖµ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.294' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.098 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v#S0´‚(:¥  ‘„ŠÓÉø³ disk.drbd1938.out‚0( disk./.usagePercent‚70.4600659978531 disk.sda1.out‚0 disk.drbd2041.out‚0 cpu‚25.164527999542635 disk.dm-0.in‚ 35.02734375 disk.drbd1931.in‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.713298161824543 disk.drbd2111.out‚0 disk.drbd1285.in‚0 network.recv‚134057 disk.sda1.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 41.515625 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1259.8975410461426 disk.drbd1510.out‚0( proc.memory.availableMB‚ 7578.8828125 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 7.427734375 network.sent‚ 1.296856e+06 disk.sda.in‚0.015625 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 35.02734375# proc.memory.rssMB‚ 2878.66015625 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.015625 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1510.in‚0$ memory_percent‚69.16144688924155 proc.cpu.threads‚17 disk.dm-0.out‚ 3933.4375 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 41.515625 disk.drbd1285.out‚0 disk.nvme0n1.out‚ 3933.4375ð¾è‚µ‚(:… ‘„ŠÓŸÏó³ gpu.0.gpu‚71.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.6' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.2 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïDÝ„“¶‚(:ˆ  „ŠÓàùѳ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚111.492' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.164 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”AnÓ6·‚(:«   „ŠÓ¦„î³ disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 cpu‚25.165366710839713 disk.drbd1965.in‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 42.15234375 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 35.171875 disk.drbd1550.out‚0 network.sent‚ 1.296922e+06 disk.sda.out‚ 42.15234375 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2878.66015625 disk.drbd1673.out‚0 disk.sda.in‚0.015625 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 network.recv‚134123 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd2041.out‚0 disk.rbd1.out‚ 7.427734375 disk.drbd1552.out‚0$ memory_percent‚69.15973027547201* proc.memory.percent‚11.713298161824543 disk.dm-0.in‚ 35.171875 disk.drbd2078.out‚0 disk.drbd2111.out‚0! disk.nvme0n1.out‚ 3933.6796875( proc.memory.availableMB‚ 7579.3046875 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1356.out‚0% disk./.usageGB‚1259.8975219726562 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.dm-0.out‚ 3933.6796875( disk./.usagePercent‚70.4600649311639 disk.drbd1817.in‚0 disk.drbd1673.in‚02¢Sà“¸‚(:ˆ ¯„ŠÓóÙ¾³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚205.089' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚68.363 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶¬ r:¹‚(:¯  ¯„ŠÓÄšÛ³# proc.memory.rssMB‚ 2879.98046875 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚3933.9169921875 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 cpu‚25.16861297088392 disk.drbd1552.in‚0 network.recv‚137021 disk.drbd1289.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 network.sent‚ 1.35951e+06 disk.drbd1817.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.drbd1938.in‚0 disk.dm-0.in‚ 38.16796875 disk.nvme0n1.in‚ 38.16796875 disk.drbd1510.in‚0( proc.memory.percent‚11.7186705271403 disk.rbd1.out‚ 7.583984375 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 43.1328125 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0% disk./.usageGB‚1259.8975219726562 disk.sda.out‚ 43.1328125 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 7577.8046875( disk./.usagePercent‚70.4600649311639 disk.sda.in‚0.015625 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda1.out‚0$ memory_percent‚69.16583379109701! disk.dm-0.out‚3933.9169921875 disk.sda1.in‚0hÌ‘º‚(:† ¾„ŠÓæÏij gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.74' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.58 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 vWV;»‚(:°  ¾„ŠÓñ³ä³ disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 7.583984375 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 cpu‚25.19209497057984 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17( proc.memory.percent‚11.7186705271403 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 network.sent‚ 1.359618e+06 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1550.in‚0% disk./.usageGB‚1259.8975410461426 disk.sda2.out‚ 44.3828125 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0! disk.dm-0.out‚3934.1591796875 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda.in‚0.015625$ memory_percent‚69.16583379109701 disk.sda2.in‚0.015625 disk.dm-0.in‚ 38.16796875 disk.drbd2118.in‚0( proc.memory.availableMB‚ 7577.8046875# proc.memory.rssMB‚ 2879.98046875 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 44.3828125 disk.drbd1673.in‚0 network.recv‚137129( disk./.usagePercent‚70.4600659978531 disk.drbd1510.out‚0 disk.drbd1552.in‚0$ disk.nvme0n1.out‚3934.1591796875 disk.drbd1510.in‚0 disk.drbd1552.out‚0)Oy2¼- _wandbruntime‚1533  _runtime‚1533ääæù/½ãt 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9916887879371643 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.23125764727592468 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.8859137296676636 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12968774139881134 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.22046156227588654 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06410256773233414 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3887307047843933 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05833183601498604 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.661873459815979 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3829595148563385 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.18049421906471252 7validation_average_jet_accuracy‚0.18018577992916107 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005666828714311123 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00845686998218298 particle/accuracy_1_of_4‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23029889166355133 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2258591502904892 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9865079522132874 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4275381565093994 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4130179286003113 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2379336804151535 jet/accuracy_4_of_4‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 % _timestamp‚1.7848407699109585e+09 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.21678008139133453 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23975209891796112 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.2939703166484833 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2001514881849289 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8944901823997498 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4115678668022156 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18803586065769196 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3461538553237915 +jet/accuracy_2_of_2‚0.07706809043884277 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.22574789822101593 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9932540059089661 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.20873601734638214 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9839285612106323 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06776305288076401 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2934948801994324 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9839285612106323 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03205128386616707 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3154561519622803 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 'particle/f_score‚0.5224654674530029 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5906944274902344 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05877231806516647 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.042543135583400726 trainer/global_step‚839 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.26432377099990845 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2670052945613861 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.271794855594635 (particle/accuracy‚0.8033855557441711 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8229761719703674 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3125666081905365 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23166970908641815 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1330157071352005 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2331618219614029 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3403990864753723 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3416537940502167 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23714230954647064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8365406394004822 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.20093156397342682 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974045753479004 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04455474764108658 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12740029394626617 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.06792058050632477 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972426295280457 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.057194311171770096 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4043390154838562 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 *jet/accuracy_1_of_3‚0.6500253081321716 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3123626410961151 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1385359913110733 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1278729885816574 *jet/accuracy_1_of_4‚0.5776154398918152 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 particle/accuracy_2_of_2‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6060605645179749 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.21528980135917664 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9992319345474243 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16409465670585632 ,jet/accuracy_3_of_3‚0.031234195455908775 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.44149908423423767 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 _step‚66 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1396609991788864 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966720938682556 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08319797366857529 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1316833645105362 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0025231081526726484 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3696470558643341 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.22406087815761566 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.052766911685466766 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012756307609379292 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17714110016822815 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17714110016822815 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4318181872367859 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23886072635650635 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22988054156303406 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874476790428162 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.506930410861969 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0050875055603682995 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6060605645179749 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.005554088857024908 +jet/accuracy_3_of_4‚0.04590190574526787 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009789139032363892 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7046142816543579 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2127797156572342 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3757118284702301 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.423174113035202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006567341275513172 +jet/accuracy_1_of_1‚0.13682366907596588 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9978319406509399 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13974983990192413 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9963547587394714 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1320972740650177 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06860049813985825 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946427941322327 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05833183601498604 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.23337514698505402 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 epoch‚2 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1371770054101944 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5636738538742065 particle/accuracy_1_of_1‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3253592550754547 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952888488769531 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.325424462556839 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33716872334480286 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.23762333393096924 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19976592063903809 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 _runtime‚ 1533.26908878 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4923834204673767 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22259625792503357 9"Purity/1FRt*SRqqt2FBt/event_puZA p rity‚0.2885590195655823 )jet/accuracy_1_of_2‚0.502668023109436 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.19848932325839996 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25 +jet/accuracy_2_of_3‚0.22049137949943542 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.44616207480430603 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05636342242360115 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.22996218502521515 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.18049421906471252 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2348063886165619 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.31405210494995117 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.36965060234069824 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22259625792503357 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 *loss/val_total_loss‚19.557257744744295 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9936938881874084 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014451040886342525 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 .particle/accuracy_4_of_4‚0.867473304271698 +particle/sensitivity‚0.5013004541397095 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9966291785240173 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.22718004882335663 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879055619239807 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002557446714490652 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_2_of_4‚0.27340078353881836 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7629175186157227 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1_(y]m:¾çt :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22259625792503357 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.2885590195655823 )jet/accuracy_1_of_2‚0.502668023109436 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.19848932325839996 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25 +jet/accuracy_2_of_3‚0.22049137949943542 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.44616207480430603 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05636342242360115 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.22996218502521515 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.18049421906471252 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2348063886165619 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.31405210494995117 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.36965060234069824 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22259625792503357 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 *loss/val_total_loss‚19.557257744744295 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9936938881874084 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014451040886342525 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 .particle/accuracy_4_of_4‚0.867473304271698 +particle/sensitivity‚0.5013004541397095 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9966291785240173 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.22718004882335663 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879055619239807 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002557446714490652 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_2_of_4‚0.27340078353881836 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7629175186157227 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9916887879371643 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.23125764727592468 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.8859137296676636 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12968774139881134 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.22046156227588654 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06410256773233414 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3887307047843933 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05833183601498604 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.661873459815979 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3829595148563385 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.18049421906471252 7validation_average_jet_accuracy‚0.18018577992916107 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005666828714311123 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00845686998218298 particle/accuracy_1_of_4‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23029889166355133 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2258591502904892 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9865079522132874 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4275381565093994 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4130179286003113 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2379336804151535 jet/accuracy_4_of_4‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 % _timestamp‚1.7848407699109585e+09 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.21678008139133453 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23975209891796112 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.2939703166484833 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2001514881849289 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8944901823997498 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4115678668022156 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18803586065769196 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3461538553237915 +jet/accuracy_2_of_2‚0.07706809043884277 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.22574789822101593 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9932540059089661 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.20873601734638214 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9839285612106323 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06776305288076401 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2934948801994324 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9839285612106323 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03205128386616707 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3154561519622803 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 'particle/f_score‚0.5224654674530029 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5906944274902344 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05877231806516647 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.042543135583400726 trainer/global_step‚839 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.26432377099990845 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2670052945613861 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.271794855594635 (particle/accuracy‚0.8033855557441711 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8229761719703674 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3125666081905365 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23166970908641815 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1330157071352005 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2331618219614029 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3403990864753723 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3416537940502167 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23714230954647064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8365406394004822 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.20093156397342682 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974045753479004 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04455474764108658 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12740029394626617 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.06792058050632477 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972426295280457 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.057194311171770096 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4043390154838562 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 *jet/accuracy_1_of_3‚0.6500253081321716 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3123626410961151 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1385359913110733 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1278729885816574 *jet/accuracy_1_of_4‚0.5776154398918152 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 particle/accuracy_2_of_2‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6060605645179749 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.21528980135917664 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9992319345474243 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16409465670585632 ,jet/accuracy_3_of_3‚0.031234195455908775 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.44149908423423767 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 _step‚66 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1396609991788864 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966720938682556 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08319797366857529 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1316833645105362 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0025231081526726484 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3696470558643341 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.22406087815761566 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.052766911685466766 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012756307609379292 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17714110016822815 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17714110016822815 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4318181872367859 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23886072635650635 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22988054156303406 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874476790428162 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.506930410861969 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0050875055603682995 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6060605645179749 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.005554088857024908 +jet/accuracy_3_of_4‚0.04590190574526787 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009789139032363892 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7046142816543579 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2127797156572342 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3757118284702301 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.423174113035202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006567341275513172 +jet/accuracy_1_of_1‚0.13682366907596588 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9978319406509399 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13974983990192413 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9963547587394714 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1320972740650177 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06860049813985825 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946427941322327 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05833183601498604 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.23337514698505402 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 epoch‚2 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1371770054101944 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5636738538742065 particle/accuracy_1_of_1‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3253592550754547 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952888488769531 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.325424462556839 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33716872334480286 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.23762333393096924 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19976592063903809 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 _runtime‚ 1533.26908878 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4923834204673767 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0Btòx“¿‚(:ˆ Í„ŠÓŸÒá³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚281.445' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.815 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?æ_4@À‚(:µ  Í„ŠÓßà€´ disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.sda2.out‚ 45.80859375" disk.dm-0.out‚4904.04931640625 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0' proc.memory.availableMB‚ 6606.421875! proc.memory.rssMB‚ 2879.984375 disk.drbd1356.out‚0% disk./.usageGB‚1260.8484001159668 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0$ memory_percent‚73.11840057373047 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd2041.out‚0% disk.nvme0n1.out‚4904.04931640625 disk.drbd1226.in‚0 disk.sda.in‚0.015625 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 7.826171875 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 network.sent‚ 1.359684e+06 network.recv‚137195 disk.drbd1584.out‚0 disk.drbd2078.in‚0 cpu‚25.190737428624498 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.718686421712238 disk.sda.out‚ 45.80859375 disk.drbd2078.out‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51324301473882U™¥2Á- _wandbruntime‚1554  _runtime‚1554ŽÅJWÆÂÀ _step‚67 ' lr-AdamW/pg1‚0.00011663265233277138 ' lr-AdamW/pg2‚0.00011663265233277138 trainer/global_step‚849 $ _timestamp‚1.784840790919379e+09 _runtime‚1554.276646619RñvÊÃÄ trainer/global_step‚849 $ _timestamp‚1.784840790919379e+09 _runtime‚1554.276646619 _step‚67 ' lr-AdamW/pg1‚0.00011663265233277138 ' lr-AdamW/pg2‚0.00011663265233277138C‚Ì2z2Ä- _wandbruntime‚1554  _runtime‚1554 'gÈÜ'ÅÖO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15284477 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7889753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68224397218304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402580992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 886521521152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 885036073762 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4861251 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1087389912064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4861510 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2269384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67136972816384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244843011233792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2591872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7899817 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15274407 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 886521521152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2269379 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7384660 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7384660 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67136604725248 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245730878191104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 886380685048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚68 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1087424401920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2592126 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245729506581504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244843011233792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7889753 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 887866957312 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7384654 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚245730878191104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244842985060352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244745278885200 _runtime‚1554.277991346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ˆPF: ÆÚO SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7899817 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15274407 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 886521521152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2269379 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7384660 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7384660 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67136604725248 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245730878191104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 886380685048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚68 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1087424401920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2592126 XIDeviceStatsMonitor.on_train_batch_start/reserved_4u&Φbytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245729506581504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244843011233792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7889753 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 887866957312 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7384654 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚245730878191104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244842985060352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244745278885200 _runtime‚1554.277991346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15284477 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7889753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68224397218304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402580992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 886521521152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 885036073762 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4861251 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1087389912064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4861510 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2269384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67136972816384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244843011233792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2591872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112D&°Ë"2Ç- _wandbruntime‚1555  _runtime‚1555xÞWJç&ÈáM trainer/global_step‚849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15291856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 887544029184 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68303903369216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚1555.757039027 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7848407923992968e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7898835 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245121920524288 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 888889472512 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67215191091200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245121894350848 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7908913 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7393021 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7393027 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245024080658976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15291856 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 887401486800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088746760704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68304305943040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7898835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245024106086944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246010809996800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7908913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245121894350848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15301940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245121920524288 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2272019 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 888889472512 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7393021 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2272024 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7393027 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2595144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1088712278016 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4866913 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246010809996800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248¶ B˜ë&ÉåM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7898835 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245121920524288 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 888889472512 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67215191091200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245121894350848 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7908913 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7393021 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7393027 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245024080658976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15291856 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 887401486800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088746760704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68304305943040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7898835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245024106086944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246010809996800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7908913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245121894350848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15301940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245121920524288 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2272019 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 888889472512 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7393021 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2272024 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7393027 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2595144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1088712278016 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4866913 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246010809996800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15291856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 887544029184 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68303903369216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚1555.757039027 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7848407923992968e+09E¤D‡\2Ê- _wandbruntime‚1555  _runtime‚1555¤U(Ëþ 1loss/FBt2/assignment_loss‚0.07590652320852835 1loss/SRqqt2/detection_loss‚0.7403953375223512 /loss/FBt2/detection_loss‚3.6300630124848214 % _timestamp‚1.7848407924352198e+09 2loss/SRqqt1/assignment_loss‚1.5128519414642523 2loss/SRqqt2/assignment_loss‚1.1979963140780814 /loss/FRt2/detection_loss‚1.4258530202356963 1loss/SRqqt1/detection_loss‚1.9812411784337411 &loss/total_loss‚20.551201571434653 epoch‚3 _step‚70 -loss/FBt1/detection_loss‚2.71510048713958 0loss/FRt1/assignment_loss‚3.8968693217115926 0loss/FRt2/assignment_loss‚2.5282997244011343 1loss/FBt1/assignment_loss‚0.22907787837017246 /loss/FRt1/detection_loss‚0.6175468323847028 trainer/global_step‚849 _runtime‚1555.792421462îÞÛÌ‚ 0loss/FRt1/assignment_loss‚3.8968693217115926 0loss/FRt2/assignment_loss‚2.5282997244011343 1loss/FBt1/assignment_loss‚0.22907787837017246 /loss/FRt1/detection_loss‚0.6175468323847028 trainer/global_step‚849 _runtime‚1555.792421462 1loss/FBt2/assignment_loss‚0.07590652320852835 1loss/SRqqt2/detection_loss‚0.7403953375223512 /loss/FBt2/detection_loss‚3.6300630124848214 % _timestamp‚1.7848407924352198e+09 2loss/SRqqt1/assignment_loss‚1.5128519414642523 2loss/SRqqt2/assignment_loss‚1.1979963140780814 /loss/FRt2/detection_loss‚1.4258530202356963 1loss/SRqqt1/detection_loss‚1.9812411784337411 &loss/total_loss‚20.551201571434653 epoch‚3 _step‚70 -loss/FBt1/detection_loss‚2.71510048713958FÚd=%>Í‚(:³  Ü„ŠÓÝé³ disk.drbd2111.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.015625 disk.sda.out‚ 46.88671875 disk.sda2.out‚ 46.88671875 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 cpu‚25.1916152112225 network.sent‚ 1.365164e+06 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1226.in‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚4904.25634765625 disk.drbd1552.out‚0 disk.drbd1289.out‚0( proc.memory.availableMB‚ 6606.1171875 proc.cpu.threads‚17 disk.dm-0.in‚ 38.16796875 disk.drbd1828.in‚0 disk.sda2.in‚0.015625# memory_percent‚73.1196403503418* proc.memory.percent‚11.718686421712238 network.recv‚138431 disk.drbd1965.out‚0 disk.rbd1.out‚ 7.826171875 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.drbd1289.in‚0! proc.memory.rssMB‚ 2879.984375) disk./.usagePercent‚70.51324301473882% disk./.usageGB‚1260.8484001159668" disk.dm-0.out‚4904.25634765625 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0,™œÒžÎ‚(:“ Ü„ŠÓ¹Ïó gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚295.859' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.61966666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïd–kžÏ‚(:“ ë„ŠÓö˜×³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚59.0 gpu.0.powerWatts‚106.13' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.376666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>5~QЂ(:­  ë„ŠÓ¼ûò³ disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 38.16796875$ disk./.usageGB‚1260.844581604004 disk.drbd2078.in‚0 disk.rbd1.out‚ 7.982421875 disk.drbd1584.out‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚5874.3642578125 disk.drbd1931.out‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 6606.2890625 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda.in‚0.015625 disk.drbd1931.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd1510.out‚0 network.sent‚ 1.365314e+06 disk.drbd2111.out‚0! proc.memory.rssMB‚ 2879.984375 proc.cpu.thr:T>çeads‚17 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.sda2.out‚47.28125* proc.memory.percent‚11.718686421712238 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0! disk.dm-0.out‚5874.3642578125 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.sda2.in‚0.015625$ memory_percent‚73.11894098917642 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 network.recv‚138581) disk./.usagePercent‚70.51302946356012 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.out‚47.28125 cpu‚25.192312303407974HÌ/H;Ñ‚(:°  ú„ŠÓÜ¿à³ disk.drbd1285.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 47.671875 disk.drbd2078.out‚0 disk.drbd1356.in‚0$ disk.nvme0n1.out‚5874.7001953125 network.sent‚ 1.36538e+06 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1226.in‚0% disk./.usageGB‚1260.8445854187012 disk.drbd2078.in‚0 disk.drbd1226.out‚0$ memory_percent‚73.12006950378418 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.718686421712238 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 47.671875 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 cpu‚25.193465977243648 disk.drbd1673.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0! proc.memory.rssMB‚ 2879.984375 disk.drbd1938.out‚0 disk.rbd1.out‚ 7.982421875 disk.drbd1510.in‚0) proc.memory.availableMB‚ 6606.01171875! disk.dm-0.out‚5874.7001953125 disk.drbd1931.out‚0 network.recv‚138647) disk./.usagePercent‚70.51302967689796 disk.sda.in‚0.015625 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0ì6±&“Ò‚(:ˆ ú„ŠÓ‹Ð¼³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚298.776' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ѧŸÓ‚(:” ‰…ŠÓ†Ë¸³ gpu.0.gpu‚63.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.757' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.91900000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó¶Ø=Ô‚(:²  ‰…ŠÓçòس disk.drbd1356.out‚0 disk.rbd1.out‚ 8.287109375 disk.nvme0n1.in‚ 38.16796875 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.015625 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0% disk./.usageGB‚1260.8445854187012 disk.drbd2078.in‚0) proc.memory.availableMB‚ 6605.27734375 disk.drbd1584.out‚0! proc.memory.rssMB‚ 2879.984375 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 network.sent‚ 1.424108e+06 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚5875.0615234375 disk.drbd1817.in‚0! disk.dm-0.out‚5875.0615234375 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 48.0546875 network.recv‚141324 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0* proc.memory.percent‚11.718686421712238 disk.sda2.out‚ 48.0546875 disk.dm-0.in‚ 38.16796875 disk.rbd1.in‚0) disk./.usagePercent‚70.51302967689796 disk.drbd1673.out‚0 disk.drbd1226.out‚0 cpu‚25.19459558935635 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625$ memory_percent‚73.12305768330893 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0L8óð“Õ‚(:ˆ ˜…ŠÓа®³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.341' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.447 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢• ½6Ö‚(:«  ˜…ŠÓçŒÑ³* proc.memory.percent‚11.718686421712238 disk.drbd2041.in‚0 disk.rbd1.out‚ 8.287109375 disk.sda1.out‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51302967689796 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.sda.in‚0.015625 disk.drbd1965.out‚0% disk./.usageGB‚1260.8445854187012 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚5875.4052734375 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0! disk.dm-0.out‚5875.4052734375$ proc.memory.availableMB‚6605.375 network.sent‚ 1.424174e+06 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 38.16796875! proc.memory.rssMB‚ 2879.984375 network.recv‚141390 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.015625$ memory_percent‚73.12266031901041 disk.drbd1226.out‚0 disk.drbd1938.in‚0 cpu‚25.19510013413012 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 48.671875 disk.sda2.out‚ 48.671875D%ÖԞׂ(:“ §…ŠÓÄÛä³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚276.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.30900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¤/í@Ø‚(:µ  §…ŠÓ³¿…´ disk.drbd1938.in‚0 disk.drbd2118.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 network.recv‚144174 disk.drbd1552.out‚0 disk.rbd1.in‚0 cpu‚25.195880402697775 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51302967689796 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 network.sent‚ 1.489001e+06 disk.drbd1817.out‚0" disk.dm-0.out‚5875.83447265625 disk.drbd2041.out‚0% disk.nvme0n1.out‚5875.83447265625 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 38.16796875 proc.cpu.threads‚17 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd2078.out‚0 disk.rbd1.out‚ 8.458984375) proc.memory.availableMB‚ 6604.95703125 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.718686421712238 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.sda.out‚ 49.0234375 disk.drbd1510.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8445854187012 disk.drbd1584.in‚0 disk.sda2.out‚ 49.0234375 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0$ memory_percent‚73.12436103820801 disk.drbd1965.out‚0 disk.drbd2078.in‚0! proc.memory.rssMB‚ 2879.984375+aÞz2Ù- _wandbruntime‚1641  _runtime‚1641¥¿ûrÅÚ¿ # _timestamp‚1.78484087770009e+09 _runtime‚1641.057327613 _step‚71 ' lr-AdamW/pg1‚0.00011654173634816238 ' lr-AdamW/pg2‚0.00011654173634816238 trainer/global_step‚899ß:+YÉÛà ' lr-AdamW/pg1‚0.00011654173634816238 ' lr-AdamW/pg2‚0.00011654173634816238 trainer/global_step‚899 # _timestamp‚1.78484087770009e+09 _runtime‚1641.057327613 _step‚71G·øu¥2Ü- _wandbruntime‚1641  _runtime‚1641ëpbÝ'Ý×O O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258685367574000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2401379 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258788475758592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8345705 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259726160662528 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8355769 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8345705 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚349 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 936139956570 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259621507530570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259726160662528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2401384 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72216575602688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 939056513536 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7803004 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 937484567856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 939056513536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7803010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71065923022848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258685393001968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2401379 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258788475758592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8345705 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259726160662528 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8355769 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8345705 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚349 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 936139956570 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 339100160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259621507530570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259726160662528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2401384 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72216575602688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 939056513536 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7803004 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 937484567856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 939056513536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7803010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71065923022848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258685393001968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653H³“LÙ2ß- _wandbruntime‚1642  _runtime‚1642ÿ¶?}o ààM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2742513 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16176242 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7811371 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259902700145352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007464077824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259067385049088 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8364865 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2404024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 $ _timestamp‚1.784840879172083e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8354787 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5146537 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7811377 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 937160758210 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚342 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259067358875648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 [HDeviceStatsMonitor.on_train_batc¨R¶Šwh_end/inactive_split_bytes.all.allocated‚72296835173888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258964169347776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259067358875648 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5146190 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 940079028736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 938733585408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16166158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8354787 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259067385049088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7811371 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71144877479936 _runtime‚1642.529655546 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7811377 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260006092461056 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2742171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258964194775744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚73 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 938733585408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71144509388800 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8364865 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 940079028736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16176242 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1151957693952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259901330105986 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260007464077824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16166158 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚347 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72296413725696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5146190 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 940079028736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 938733585408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16166158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8354787 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259067385049088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 339100160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7811371 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71144877479936 _runtime‚1642.529655546 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7811377 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260006092461056 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2742171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258964194775744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚73 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 938733585408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71144509388800 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8364865 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 940079028736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16176242 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1151957693952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259901330105986 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260007464077824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16166158 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚347 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72296413725696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16176242 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7811371 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1395809792 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259902700145352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007464077824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259067385049088 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8364865 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2404024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 $ _timestamp‚1.784840879172083e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8354787 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5146537 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7811377 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 937160758210 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚342 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259067358875648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72296835173888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056I÷±Õ2â- _wandbruntime‚1642  _runtime‚1642›tÒ*ãû 0loss/FRt2/assignment_loss‚2.6101114980791915 1loss/FBt2/assignment_loss‚0.08851226639934051 /loss/FRt1/detection_loss‚0.5906802423119336 trainer/global_step‚899 % _timestamp‚1.7848408792121985e+09 _step‚74 0loss/FBt1/assignment_loss‚0.5270687805135704 .loss/FBt1/detection_loss‚13.86283350195138 epoch‚3 2loss/SRqqt1/assignment_loss‚1.3638472683668375 1loss/SRqqt2/assignment_loss‚1.108377887906725 /loss/FBt2/detection_loss‚1.6415160827358783 %loss/total_loss‚29.81538331790633 _runtime‚1642.569397638 0loss/FRt1/assignment_loss‚3.8577503980364707 /loss/FRt2/detection_loss‚1.4296020059213108 0loss/SRqqt1/detection_loss‚2.007407562040067 1loss/SRqqt2/detection_loss‚0.7276758236436284íWl{äÿ epoch‚3 2loss/SRqqt1/assignment_loss‚1.3638472683668375 1loss/SRqqt2/assignment_loss‚1.108377887906725 /loss/FBt2/detection_loss‚1.6415160827358783 %loss/total_loss‚29.81538331790633 _runtime‚1642.569397638 0loss/FRt1/assignment_loss‚3.8577503980364707 /loss/FRt2/detection_loss‚1.4296020059213108 0loss/SRqqt1/detection_loss‚2.007407562040067 1loss/SRqqt2/detection_loss‚0.7276758236436284 0loss/FRt2/assignment_loss‚2.6101114980791915 1loss/FBt2/assignment_loss‚0.08851226639934051 /loss/FRt1/detection_loss‚0.5906802423119336 trainer/global_step‚899 % _timestamp‚1.7848408792121985e+09 _step‚74 0loss/FBt1/assignment_loss‚0.5270687805135704 .loss/FBt1/detection_loss‚13.86283350195138J4+³¤å‚(:’ ¶…ŠÓ߀â³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚322.951' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚107.65033333333334 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&‹V¦4æ‚(:©  ¶…ŠÓžñ‚´% disk./.usageGB‚1260.8445854187012 disk.drbd1938.in‚0" disk.dm-0.out‚5876.02587890625 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0$ proc.memory.availableMB‚6604.875! proc.memory.rssMB‚ 2879.984375 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.015625 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0% disk.nvme0n1.out‚5876.02587890625 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0$ memory_percent‚73.12469482421875 network.sent‚ 1.489067e+06 disk.sda2.in‚0.015625 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.718686421712238 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 8.458984375 network.recv‚144240 disk.sda.out‚49.3125 disk.drbd1965.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 cpu‚25.19651167259307) disk./.usagePercent‚70.51302967689796 disk.sda2.out‚49.3125 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0‡5m-<ç‚(:±  Å…ŠÓ«‚ý³ disk.drbd1938.out‚0 disk.drbd1356.in‚0" disk.nvme0n1.out‚ 5876.32421875 disk.drbd1226.out‚0* proc.memory.percent‚11.718686421712238 network.sent‚ 1.489175e+06) disk./.usagePercent‚70.51302967689796 disk.sda2.out‚ 49.67578125 disk.dm-0.out‚ 5876.32421875 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0! proc.memory.rssMB‚ 2879.984375 disk.drbd1828.out‚0 disk.sda.in‚0.015625 disk.rbd1.in‚0 disk.sda2.in‚0.015625 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 7484.4453125% disk./.usageGB‚1260.8445854187012 disk.drbd1356.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0$ memory_percent‚69.54571406046549 disk.drbd1938.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.rbd1.out‚ 8.4912109375 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 network.recv‚144348 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1550.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 49.67578125 disk.drbd1673.in‚0 disk.drbd2078.out‚0 cpu‚25.197279723689174 proc.cpu.threads‚17ÖÑ.žè‚(:“ Å…ŠÓÆ÷ܳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.46' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.48666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M¬ mŸé‚(:” Ô…ŠÓ׬߳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚164.813' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚54.937666666666665 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðù¯R=ê‚(:²  Ô…ŠÓø§´) proc.memory.availableMB‚ 7187.67578125! proc.memory.rssMB‚ 2879.984375 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 5876.49609375 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0" disk.nvme0n1.out‚ 5876.49609375) disk./.usagePercent‚70.51303074358717 disk.drbd1552.out‚0 disk.sda2.in‚0.015625 network.sent‚ 1.494689e+06 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 8.4912109375 disk.dm-0.in‚ 38.16796875 network.recv‚145572 disk.sda.out‚ 50.02734375 disk.sda1.out‚0 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 cpu‚25.198177910953277 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.sda.in‚0.015625 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.718686421712238 disk.nvme0n1.in‚ 38.16796875 disk.drbd1510.in‚0% disk./.usageGB‚1260.8446044921875 disk.sda2.out‚ 50.02734375 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0$ memory_percent‚70.75327237447104 proc.cpu.threads‚17 MV9ë‚(:®  ã…ŠÓ«å³ disk.drbd1965.in‚0 disk.drbd1828.out‚0& proc.memory.availableMB‚ 6883.03125 disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51302967689796% disk./.usageGB‚1260.8445854187012$ disk.nvme0n1.out‚5876.7509765625 disk.sda1.out‚0 disk.dm-0.in‚ 38.16796875 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0! disk.dm-0.out‚5876.7509765625 disk.rbd1.out‚ 8.6474609375! proc.memory.rssMB‚ 2879.984375 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 cpu‚25.19863431620212 disk.drbd2118.in‚0 disk.drbd2111.out‚0 network.recv‚145722 disk.drbd1285.in‚0 disk.sda.in‚0.015625$ memory_percent‚71.99287414550781 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 network.sent‚ 1.494839e+06 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 50.796875 disk.drbd1938.out‚0* proc.memory.percent‚11.718686421712238 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.sda.out‚ 50.796875 disk.drbd1828.in‚0×./Õ“ì‚(:ˆ ã…ŠÓ²´Æ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.477' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.159 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äæ”í‚(:‰ ò…ŠÓýƒä³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.698' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Ð2;î‚(:°  ò…ŠÓ•Àƒ´ disk.drbd1828.in‚0% disk.nvme0n1.out‚5877.02880859375 disk.drbd1931.out‚0$ disk./.usageGB‚1260.844612121582 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0" disk.dm-0.out‚5877.02880859375 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 51.0703125 cpu‚25.19935567718138 disk.drbd2041.out‚0 network.sent‚ 1.559751e+06 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0 network.recv‚148756 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0( proc.memory.availableMB‚ 6585.6171875 disk.drbd1817.out‚0 disk.sda2.in‚0.015625$ memory_percent‚73.20305506388345 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 8.6474609375 disk.dm-0.in‚ 38.16796875* proc.memory.percent‚11.715189615885418 proc.memory.rssMB‚2879.125) disk./.usagePercent‚70.51303117026285 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 51.0703125 µàžï‚(:“ †ŠÓ÷ù¼³ gpu.0.gpu‚45.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.961' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.32033333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†rt;ð‚(:°  †ŠÓÌžÚ³ disk.drbd2078.out‚0% disk.nvme0n1.out‚5877.71044921875 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1938.out‚0" disk.dm-0.out‚5877.71044921875 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 51.5390625 disk.sda2.in‚0.015625 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 proc.memory.rssMB‚2879.125 network.recv‚149864) disk./.usagePercent‚70.51304269050625 disk.rbd1.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 6280.95703125 disk.drbd1828.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 network.sent‚ 1.565506e+06 disk.rbd1.out‚ 8.6796875* proc.memory.percent‚11.715189615885418 cpu‚25.200223837195512 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.sda.in‚0.015625 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 38.16796875 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd2041.in‚0$ memory_percent‚74.44272041320801 disk.drbd1510.in‚0 disk.sda2.out‚ 51.5390625% disk./.usageGB‚1260.8448181152344 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0iKÚ2ñ- _wandbruntime‚1728  _runtime‚1728®u©ÇòÁ ' lr-AdamW/pg2‚0.00011644322087770242 trainer/global_step‚949 % _timestamp‚1.7848409648144712e+09 _runtime‚1728.171750146 _step‚75 ' lr-AdamW/pg1‚0.00011644322087770242ŠÔ×äËóÅ ' lr-AdamW/pg1‚0.00011644322087770242 ' lr-AdamW/pg2‚0.00011644322087770242 trainer/global_step‚949 % _timestamp‚1.7848409648144712e+09 _runtime‚1728.171750146 _step‚75KõÒqÙ2ô- _wandbruntime‚1728  _runtime‚1728æKÄpÜõ×O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273722814743552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 988900633600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273724186353152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_poolÌýNW".current‚352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272625456262800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1212915843072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273614070141432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8221360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8221354 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2886544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8801657 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2533384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 990246069760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273612700102178 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17033081 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 988900633600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272733914109952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2886896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272625481690768 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 988588450664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 987243839378 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74995609411584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273722814743552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272733940283392 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5420280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚357 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1212973401600 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8801657 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272733940283392 _step‚76 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2533379 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248›‚ñ¾á'öÛO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1212915843072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273614070141432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8221360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8221354 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2886544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8801657 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2533384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 990246069760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273612700102178 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17033081 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 988900633600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272733914109952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2886896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272625481690768 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 988588450664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 987243839378 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74995609411584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273722814743552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272733940283392 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5420280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚357 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1212973401600 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8801657 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272733940283392 _step‚76 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2533379 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273722814743552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 988900633600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273724186353152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272625456262800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚949LQ®ˆ2÷- _wandbruntime‚1729  _runtime‚1729mç&øáM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272904258036576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75073827686400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76288409200128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17050544 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 989609252416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76287989849088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 988264641018 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274004118158848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5425832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2889455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8810739 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273012849573888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8229721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848409662947075e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274002746542080 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8810739 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2536024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2536019 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273012849573888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5425474 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273892522677594 _step‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75074195777536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8229727 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17040460 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1214213422592 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274004118158848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚358 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1214162162688 _runtime‚1729.652326445 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 989923141632 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17050544 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8820817 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273012823400448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8820817 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 991268584960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚273012823400448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273893892716960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17040460À àðâùåM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17040460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272904258036576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75073827686400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76288409200128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17050544 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 989609252416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76287989849088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 988264641018 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274004118158848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5425832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2889455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4751517>Ãçd 2864 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8810739 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273012849573888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8229721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848409662947075e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274002746542080 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8810739 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2536024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2536019 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273012849573888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5425474 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273892522677594 _step‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75074195777536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8229727 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17040460 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1214213422592 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274004118158848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚358 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1214162162688 _runtime‚1729.652326445 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 989923141632 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17050544 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8820817 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273012823400448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8820817 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 991268584960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚273012823400448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273893892716960MÓZÄÿ2ú- _wandbruntime‚1729  _runtime‚1729ÜÓ ûý 2loss/SRqqt2/assignment_loss‚1.0458563080713104 /loss/FRt1/detection_loss‚0.6713660109665165 /loss/FRt2/detection_loss‚1.5555538807991371 0loss/SRqqt1/detection_loss‚2.223781348649727 1loss/SRqqt2/detection_loss‚0.7722885440712695 epoch‚3 /loss/FRt2/assignment_loss‚2.326858141950805 1loss/FBt1/assignment_loss‚0.38862919246633376 1loss/FBt2/assignment_loss‚0.10890593823710379 .loss/FBt1/detection_loss‚2.774722173296502 /loss/FBt2/detection_loss‚1.8244191785670227 trainer/global_step‚949 _runtime‚1729.687085585 _step‚78 0loss/FRt1/assignment_loss‚3.6691022304450573 2loss/SRqqt1/assignment_loss‚1.4133408988911318 &loss/total_loss‚18.774823846411916 % _timestamp‚1.7848409663298614e+09Þïü &loss/total_loss‚18.774823846411916 % _timestamp‚1.7848409663298614e+09 2loss/SRqqt2/assignment_loss‚1.0458563080713104 /loss/FRt1/detection_loss‚0.6713660109665165 /loss/FRt2/detection_loss‚1.5555538807991371 0loss/SRqqt1/detection_loss‚2.223781348649727 1loss/SRqqt2/detection_loss‚0.7722885440712695 epoch‚3 /loss/FRt2/assignment_loss‚2.326858141950805 1loss/FBt1/assignment_loss‚0.38862919246633376 1loss/FBt2/assignment_loss‚0.10890593823710379 .loss/FBt1/detection_loss‚2.774722173296502 /loss/FBt2/detection_loss‚1.8244191785670227 trainer/global_step‚949 _runtime‚1729.687085585 _step‚78 0loss/FRt1/assignment_loss‚3.6691022304450573 2loss/SRqqt1/assignment_loss‚1.4133408988911318N´¹–”ý‚(:‰ †ŠÓÿýá³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚320.379' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚106.793 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UYç)þ‚(:ž  †ŠÓ¬ÿ³ disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 38.16796875 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8442459106445 disk.drbd1817.out‚0 disk.sda2.out‚52 disk.drbd2041.out‚0( proc.memory.availableMB‚ 5921.7109375 disk.nvme0n1.in‚ 38.16796875 disk.drbd1931.out‚0$ memory_percent‚75.90449651082358 disk.sda1.in‚0 disk.sda2.in‚0.015625 disk.drbd1552.out‚0 network.recv‚149930 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚5878.24169921875 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.715189615885418 disk.rbd1.out‚ 8.6796875 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0% disk.nvme0n1.out‚5878.24169921875 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.sda.out‚52 disk.drbd1938.out‚0 disk.drbd2111.out‚0 network.sent‚ 1.565572e+06 proc.memory.rssMB‚2879.125 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.015625 disk.drbd2078.out‚0 disk.drbd1285.out‚0 cpu‚25.20049535164554) disk./.usagePercent‚70.51301068983012 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0ü(ZD“ÿ‚(:ˆ Ÿ†ŠÓ„´Ð³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚245.856' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚81.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^–p-?€‚(:´  Ÿ†ŠÓ¾¾î³ disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 38.16796875 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8452110290527 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.015625" disk.dm-0.out‚5879.20751953125 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0 cpu‚25.20119381726121 proc.cpu.threads‚17 network.recv‚150879 disk.drbd1828.in‚0% disk.nvme0n1.out‚5879.20751953125 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 52.20703125 disk.drbd1289.in‚0* proc.memory.percent‚11.718686421712238 disk.drbd1552.out‚0$ memory_percent‚77.07114219665527) proc.memory.availableMB‚ 5634.99609375 disk.drbd1510.out‚0 disk.rbd1.in‚0 network.sent‚ 1.568586e+06 disk.drbd1356.out‚0 disk.drbd1673.in‚0! proc.memory.rssMB‚ 2879.984375 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 8.8359375 disk.nvme0n1.in‚ 38.16796875 disk.sda2.out‚ 52.20703125 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.015625) disk./.usagePercent‚70.51306466430385 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0íâž‚(:“ ®†ŠÓχµ³ gpu.0.gpu‚31.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.76' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.91999999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y-$">‚‚(:³  ®†ŠÓÓÓÒ³ disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.sda.in‚0.015625 disk.drbd2111.out‚0) proc.memory.availableMB‚ 5635.33203125 network.sent‚ 1.568652e+06 network.recv‚150945 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51306551765522 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0$ memory_percent‚77.06977526346842 proc.cpu.threads‚17 disk.drbd1938.out‚0 cpu‚25.202160506467603 disk.drbd1828.in‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2879.12890625% disk./.usageGB‚1260.8452262878418 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.715205510457356 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 8.8359375% disk.nvme0n1.out‚5879.67333984375 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.sda2.out‚ 52.59765625 disk.drbd1938.in‚0 disk.dm-0.in‚ 41.390625 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.sda.out‚ 52.59765625 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 41.390625 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0" disk.dm-0.out‚5879.67333984375 disk.drbd2111.in‚0³ãxL3ƒ‚(:¨  ½†ŠÓ¨òç³ disk.drbd1673.out‚0 disk.drbd2078.out‚0) disk./.usagePercent‚70.51306445096603 disk.drbd2111.in‚0 disk.rbd1.out‚ 8.8681640625 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 cpu‚25.20255028276666 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 network.recv‚151053 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 44.078125 disk.sda2.in‚0.015625 disk.sda2.out‚ 52.9140625 disk.drbd1931.in‚0 proc.cpu.threads‚17 network.sent‚ 1.56876e+06 disk.drbd1510.out‚0 disk.drbd1938.out‚0$ memory_percent‚77.06979115804037 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2879.12890625 disk.drbd1552.out‚0 disk.sda.in‚0.015625 disk.drbd1356.out‚0 disk.drbd2118.out‚0 disk.drbd1828.out‚0! disk.nvme0n1.out‚ 5880.6328125 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8452072143555 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0' proc.memory.availableMB‚ 5635.328125 disk.sda1.out‚0 disk.drbd1285.in‚0 disk.dm-0.out‚ 5880.6328125 disk.drbd1226.in‚0 disk.sda.out‚ 52.9140625 disk.drbd1289.out‚0 disk.dm-0.in‚ 44.078125* proc.memory.percent‚11.715205510457356 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 ¨ޟ„‚(:” ½†ŠÓˬʳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.676' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.89200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ão52Ÿ…‚(:” ̆ŠÓô»·³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.001' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.00033333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›få-†‚(:¢  ̆ŠÓƒâÙ³ network.recv‚153329% disk./.usageGB‚1260.8452072143555 disk.sda.out‚53.25 disk.nvme0n1.in‚ 44.078125# proc.memory.rssMB‚ 2879.12890625 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.dm-0.out‚ 5880.79296875 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 network.sent‚ 1.630849e+06 disk.drbd1584.in‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.715205510457356 cpu‚25.203077958946658 disk.drbd1552.out‚0 disk.sda2.out‚53.25 disk.drbd2111.out‚0" disk.nvme0n1.out‚ 5880.79296875 disk.drbd1226.out‚0' proc.memory.availableMB‚ 5635.328125 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 8.8681640625 disk.sda.in‚0.015625 disk.drbd1817.in‚0) disk./.usagePercent‚70.51306445096603 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0$ memory_percent‚77.06979115804037 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 44.078125 disk.drbd1356.out‚0 disk.rbd1.in‚0²0/Ÿ‡‚(:” Û†ŠÓò𮳠gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚111.344' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.114666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤€b§=ˆ‚(:²  Û†ŠÓ̳̃ disk.nvme0n1.in‚ 44.078125 disk.drbd2118.out‚0" disk.dm-0.out‚5881.41162109375 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 cpu‚25.203732218764944 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 53.61328125 network.sent‚ 1.630957e+06% disk./.usageGB‚1260.8452072143555 disk.sda1.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.sda.in‚0.015625 disk.dm-0.in‚ 44.078125 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚5881.41162109375 disk.drbd1552.in‚0$ memory_percent‚77.07328796386719 disk.drbd1510.in‚0 disk.drbd1356.out‚0 network.recv‚153437 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 9.0244140625) proc.memory.percent‚11.71870231628418 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.51306445096603 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 53.61328125 disk.drbd1965.out‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2879.98828125 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 5634.46875 disk.drbd2041.in‚0 disk.drbd2041.out‚0~-2‰- _wandbruntime‚1815  _runtime‚1815šò_'ÆŠÀ trainer/global_step‚999 % _timestamp‚1.7848410518501365e+09 _runtime‚ 1815.20743154 _step‚79 ' lr-AdamW/pg1‚0.00011633711883451256 ' lr-AdamW/pg2‚0.00011633711883451256^¯š¸Ê‹Ä ' lr-AdamW/pg2‚0.00011633711883451256 trainer/global_step‚999 % _timestamp‚1.7848410518501365e+09 _runtime‚ 1815.20743154 _step‚79 ' lr-AdamW/pg1‚0.00011633711883451256Oâ´µÌ2Œ- _wandbruntime‚1815  _runtime‚1815wq¯dâ'ÜO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 341916160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17897313 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8639710 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286679378634752 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287719468824576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1041435625984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286565544951600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1041435625984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3034652 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80200148896256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286565570379568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1038347722186 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8639710 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287720840434176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1404197376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286679378634752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17907383 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286679404808192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8639704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8639704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9257609 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1039692333472 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9257609 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286679404808192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1275221187072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78924927709184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78924559618048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1040090189824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚287720840434176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3034295 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 $ _timestamp‚1.784841051851313e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2665384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287603892673786 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9267673 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287605262713040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚287719468824576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1040090189824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5699674 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5700036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2665379 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1275174114304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ö²fŽàO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287720840434176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1404197376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286679378634752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17907383 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286679404808192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8639704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8639704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9257609 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1039692333472 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9257609 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286679404808192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1275221187072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78924927709184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.laƒ—€rge_pool.freed‚78924559618048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1040090189824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚287720840434176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3034295 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 $ _timestamp‚1.784841051851313e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2665384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚609 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287603892673786 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9267673 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287605262713040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚287719468824576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1040090189824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5699674 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5700036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2665379 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1275174114304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 341916160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17897313 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8639710 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286679378634752 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287719468824576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1041435625984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286565544951600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1041435625984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3034652 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80200148896256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286565570379568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1038347722186 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8639710 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚641 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005Pæߌ°2- _wandbruntime‚1816  _runtime‚1816JΞì&æM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288000772239872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287999400623104 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1041112697856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286958314098688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1040713135224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286958287925248 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9276769 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17924846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚999 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5705637 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288000772239872 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚287883715249202 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8648071 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2668024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9276769 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9266691 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80279994418688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17914762 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287885085288568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1041112697856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79003514075136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648071 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚1816.688441765 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1276426986496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80279572970496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1404197376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1039368523826 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _step‚81 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79003145984000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286844372153344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5705275 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1042458141184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2668019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286844346725376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9266691 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286958287925248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1042458141184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17914762 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440""f-ð&‘êM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17914762 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287885085288568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1041112697856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79003514075136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648071 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚1816.688441765 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1276426986496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚641 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80279572970496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1404197376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1039368523826 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _step‚81 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79003145984000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286844372153344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5705275 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1042458141184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2668019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚609 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286844346725376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9266691 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286958287925248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1042458141184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17914762 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288000772239872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287999400623104 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1041112697856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286958314098688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1040713135224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286958287925248 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9276769 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17924846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚999 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5705637 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288000772239872 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚287883715249202 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8648071 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2668024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9276769 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9266691 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80279994418688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Q÷V2’- _wandbruntime‚1816  _runtime‚1816Hq¬@“ý /loss/FBt2/detection_loss‚2.1046775555676147 trainer/global_step‚999 0loss/FRt1/assignment_loss‚3.5077897871993735 0loss/FBt1/assignment_loss‚0.4639673803241984 1loss/FBt2/assignment_loss‚0.12662655767116857 1loss/SRqqt1/detection_loss‚1.7283980703043311 /loss/FBt1/detection_loss‚2.7516854074940116 /loss/FRt2/assignment_loss‚2.089551401687852 2loss/SRqqt1/assignment_loss‚1.2004509718862135 2loss/SRqqt2/assignment_loss‚1.0041544075986484 /loss/FRt1/detection_loss‚0.6627200978658842 /loss/FRt2/detection_loss‚1.5596060265369172 1loss/SRqqt2/detection_loss‚0.8448574296063807 epoch‚3 $ _timestamp‚1.784841053366504e+09 &loss/total_loss‚18.044485093742594 _runtime‚1816.723692139 _step‚82ðz]Š” 0loss/FRt1/assignment_loss‚3.5077897871993735 0loss/FBt1/assignment_loss‚0.4639673803241984 1loss/FBt2/assignment_loss‚0.12662655767116857 1loss/SRqqt1/detection_loss‚1.7283980703043311 /loss/FBt1/detection_loss‚2.7516854074940116 /loss/FRt2/assignment_loss‚2.089551401687852 2loss/SRqqt1/assignment_loss‚1.2004509718862135 2loss/SRqqt2/assignment_loss‚1.0041544075986484 /loss/FRt1/detection_loss‚0.6627200978658842 /loss/FRt2/detection_loss‚1.5596060265369172 1loss/SRqqt2/detection_loss‚0.8448574296063807 epoch‚3 $ _timestamp‚1.784841053366504e+09 &loss/total_loss‚18.044485093742594 _runtime‚1816.723692139 _step‚82 /loss/FBt2/detection_loss‚2.1046775555676147 trainer/global_step‚999Ró&Íž•‚(:“ ꆊÓ´¹°³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.337' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.44566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M/ {:–‚(:¯  ꆊÓÑ×ʳ disk.drbd1817.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.714633305867512 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 44.078125 network.sent‚ 1.631023e+06 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.sda.out‚ 53.984375 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0$ memory_percent‚77.06842422485352) disk./.usagePercent‚70.51306445096603 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 cpu‚25.204641628544028 disk.drbd1552.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 44.078125 disk.sda2.out‚ 53.984375 disk.drbd1828.out‚0 disk.rbd1.out‚ 9.0244140625 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2878.98828125% disk./.usageGB‚1260.8452072143555 disk.sda2.in‚0.015625 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.sda.in‚0.015625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0! disk.dm-0.out‚5881.6943359375 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚5881.6943359375( proc.memory.availableMB‚ 5635.6640625 disk.drbd1965.in‚0 disk.drbd1510.in‚0 network.recv‚153503y¤Zž—‚(:“ ù†ŠÓùçɳ gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.849' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.94966666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=?Ìs˜‚(:µ  ù†ŠÓöíì³ disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 9.056640625% disk./.usageGB‚1260.8452072143555 disk.sda.in‚0.015625 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚5882.51806640625) proc.memory.availableMB‚ 5635.04296875 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2879.12890625 disk.drbd1817.out‚0 disk.sda.out‚ 54.28515625 disk.drbd1931.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.715205510457356 disk.drbd1550.out‚0 disk.drbd1226.d$Íin‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd1356.in‚0 disk.drbd2111.out‚0 cpu‚25.205395416286837 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1828.out‚0 network.recv‚154861) disk./.usagePercent‚70.51306445096603 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 54.28515625 disk.nvme0n1.in‚ 44.078125 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 44.078125 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 network.sent‚ 1.639148e+06 disk.drbd2118.in‚0 disk.drbd1965.out‚0$ memory_percent‚77.07095146179199 disk.drbd1931.in‚0" disk.dm-0.out‚5882.51806640625 disk.drbd1965.in‚0 disk.drbd1285.in‚0ëYÔRž™‚(:“ ˆ‡ŠÓÄèÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.107' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.70233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fŸv<š‚(:±  ˆ‡ŠÓàó³ disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 44.078125 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.rbd1.out‚ 9.236328125% disk.nvme0n1.out‚5882.70556640625 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.sda2.out‚ 54.77734375 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda.out‚ 54.77734375" disk.dm-0.out‚5882.70556640625 disk.drbd1673.in‚0 disk.drbd1931.in‚0 cpu‚25.205747365625978 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.sda2.in‚0.015625* proc.memory.percent‚11.715205510457356 network.sent‚ 1.639256e+06) disk./.usagePercent‚70.51306445096603 disk.drbd1938.in‚0 disk.drbd1552.in‚0$ memory_percent‚77.07290649414062 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0% proc.memory.availableMB‚ 5634.5625 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2879.12890625 network.recv‚154969% disk./.usageGB‚1260.8452072143555 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.015625 disk.drbd1673.out‚0 disk.dm-0.in‚ 44.078125 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0!›‚(:“ —‡ŠÓâ¼å³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚255.835' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.27833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²M6ë9œ‚(:®  —‡ŠÓÌׇ´ disk.drbd1356.out‚0 disk.sda1.in‚0) proc.memory.percent‚11.71870231628418 cpu‚25.207007584118166 disk.dm-0.in‚ 44.69921875 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2879.98828125 disk.drbd2041.in‚0 disk.sda2.out‚55.21875 network.recv‚155035 disk.drbd2111.in‚0 disk.drbd1550.out‚0! disk.dm-0.out‚5883.1650390625 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 9.384765625 disk.drbd1552.out‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51306679768227 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 44.69921875 disk.drbd1817.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0$ disk.nvme0n1.out‚5883.1650390625 disk.drbd1584.out‚0 disk.sda.out‚55.21875 disk.drbd1673.in‚0$ memory_percent‚77.07640329996744 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1289.in‚0 network.sent‚ 1.639322e+06 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 5633.703125% disk./.usageGB‚1260.8452491760254 disk.drbd1965.in‚0 disk.drbd1673.out‚0VÅ—ß?‚(:´  ¦‡ŠÓ›Úг disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0$ memory_percent‚77.07651456197104 disk.drbd2118.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2879.98828125$ disk./.usageGB‚1260.845230102539 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51306573099306 disk.dm-0.in‚ 44.69921875! disk.dm-0.out‚5883.4501953125 disk.sda.out‚ 55.56640625 proc.cpu.threads‚17 disk.drbd1356.out‚0$ disk.nvme0n1.out‚5883.4501953125 disk.sda2.in‚0.015625 disk.sda2.out‚ 55.56640625 disk.drbd1931.in‚0 network.sent‚ 1.704238e+06 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.sda.in‚0.015625 disk.drbd1285.out‚0 disk.drbd2078.out‚0 network.recv‚157941 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 cpu‚25.20748460040436 disk.drbd2041.in‚0 disk.rbd1.out‚ 9.384765625 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 5633.67578125 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0) proc.memory.percent‚11.71870231628418 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 44.69921875V Å.“ž‚(:ˆ ¦‡ŠÓ©í®³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚110.055' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó˜ñ2Ÿ- _wandbruntime‚1902  _runtime‚1902´³’‡È Â ' lr-AdamW/pg1‚0.00011622344412613976 ' lr-AdamW/pg2‚0.00011622344412613976 trainer/global_step‚1049 % _timestamp‚1.7848411387308633e+09 _runtime‚1902.088835106 _step‚83'&;Ì¡Æ _runtime‚1902.088835106 _step‚83 ' lr-AdamW/pg1‚0.00011622344412613976 ' lr-AdamW/pg2‚0.00011622344412613976 trainer/global_step‚1049 % _timestamp‚1.7848411387308633e+09S·eý2¢- _wandbruntime‚1902  _runtime‚1902pµO«ä'£ÞO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2797379 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1337278961664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1091279746048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1091279746048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301717494515200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300624843159552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚381 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9058054 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300505633640400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18771615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3182527 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1089451604994 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚1902.089104399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9713561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848411387316964e+09 _step‚84 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1092625182208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2797384 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9713561 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9723625 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300624843159552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9058054 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300624869332992 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84191156877312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9723625 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚301717494515200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1092625182208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9058060 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301716122905600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300624869332992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300505659068368 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84191578332672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82853877915648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18781685 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9058060 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301716122905600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300624869332992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300505659068368 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84191578332672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82853877915648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18781685 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9058060 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1091279746048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1091279746048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301717494515200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300624843159552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚381 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9058054 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300505633640400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18771615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3182527 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1089451604994 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚1902.089104399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9713561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848411387316964e+09 _step‚84 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1092625182208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2797384 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9713561 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9723625 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300624843159552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9058054 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300624869332992 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84191156877312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9723625 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864TœÌoõ2¥- _wandbruntime‚1903  _runtime‚1903ØÿêúQ¦èM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301996054704128 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9732721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82932832372736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338527465472 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9732721 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18799148 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2800019 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3185864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301874907820810 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18789064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚1903.562709438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1092302254080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18789064 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2800024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1091817018032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9066427 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300903778623488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300903778623488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301997426320896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3185494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9722643 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9066421 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84270991747072 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300784460842144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82932464281600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301997426320896 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18799148 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300903752450048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5985513 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84271411098112 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 % _timestamp‚1.7848411402051084e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9066421 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301876277860176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 trainer/global_step‚1049 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9066427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181–ÌåÒò&§ìM WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1092302254080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18789064 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2800024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1091817018032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9066427 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300903778623488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300903778623488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301997426320896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3185494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9722643 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9066421 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84270991747072 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300784460842144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82932464281600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301997426320896 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18799148 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300903752450048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5985513 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84271411098112 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 % _timestamp‚1.7848411402051084e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9066421 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301876277860176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 trainer/global_step‚1049 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9066427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301996054704128 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9732721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82932832372736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338527465472 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9732721 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18799148 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2800019 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3185864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301874907820810 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18789064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚1903.562709438 Eµ‚(:³  €ˆŠÓûÅ…´ disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 9.60546875% disk.nvme0n1.out‚5887.66064453125 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0" disk.dm-0.out‚5887.66064453125 disk.drbd2118.in‚0 disk.sda2.out‚ 57.5390625 disk.drbd1965.out‚0* proc.memory.percent‚11.718734105428059 disk.sda2.in‚0.015625 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.sda.out‚ 57.5390625 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0& proc.memory.availableMB‚ 5633.46875# proc.memory.rssMB‚ 2879.99609375 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 44.78515625 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 network.sent‚ 1.775233e+06 disk.drbd2118.out‚0 disk.drbd1285.in‚0 network.recv‚162338% disk./.usageGB‚1260.8453979492188 disk.drbd1584.out‚0 disk.drbd1965.in‚0 cpu‚25.209994770939073 disk.drbd1817.out‚0 disk.sda.in‚0.015625 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0) disk./.usagePercent‚70.51307511785807 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1828.out‚0$ memory_percent‚77.07735697428384 disk.nvme0n1.in‚ 44.78515625 disk.drbd1226.out‚0bt¥Ú“¶‚(:ˆ €ˆŠÓðøà³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.118' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.706 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø‚úò2·- _wandbruntime‚1989  _runtime‚1989 F›È¸Â ' lr-AdamW/pg2‚0.00011610221165273387 trainer/global_step‚1099 % _timestamp‚1.7848412256446347e+09 _runtime‚1989.001866038 _step‚87 ' lr-AdamW/pg1‚0.00011610221165273387,Ó%Ì¹Æ ' lr-AdamW/pg1‚0.00011610221165273387 ' lr-AdamW/pg2‚0.00011610221165273387 trainer/global_step‚1099 % _timestamp‚1.7848412256446347e+09 _runtime‚1989.001866038 _step‚87WZv¶Œ2º- _wandbruntime‚1989  _runtime‚19897)-¥‚ »âO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315712776986624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9476404 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3330710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315714148596224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315712776986624 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2929384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚380 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314570333857792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19645917 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.Ð9å{fsmall_pool.allocated‚ 1143814738432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3331090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9476410 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1140555487802 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88183056510976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10179577 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6260089 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315714148596224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86783564304384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143814738432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88182635055616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314570333857792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399438842368 trainer/global_step‚1099 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1142469302272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 % _timestamp‚1.7848412256456952e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10179577 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315587647856256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10169513 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314570307684352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10169513 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚385 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚88 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600Žòm¬ì'¼æO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314570333857792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399438842368 trainer/global_step‚1099 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1142469302272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 % _timestamp‚1.7848412256456952e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10179577 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315587647856256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10169513 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314570307684352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10169513 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚385 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚88 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315712776986624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9476404 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3330710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315714148596224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315712776986624 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2929384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚380 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314570333857792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19645917 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1143814738432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3331090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9476410 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1140555487802 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88183056510976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10179577 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6260089 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315714148596224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86783564304384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143814738432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88182635055616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653X.P}‘2½- _wandbruntime‚1990  _runtime‚1990ä"-Yò&¾ìM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6265691 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88262467935744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315994080401920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88262893578240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315867470431784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314849216974848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848412271158109e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19673450 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2932019 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86861782579200 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9484771 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚89 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3333672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10178595 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314849216974848 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315866100392418 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86862150670336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1099 _runtime‚1990.473393923 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1143491810304 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0»[ øg¿ðM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10178595 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314849216974848 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315866100392418 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86862150670336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1099 _runtime‚1990.473393923 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1143491810304 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6265691 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88262467935744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315994080401920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88262893578240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315867470431784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314849216974848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848412271158109e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19673450 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2932019 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86861782579200 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9484771 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚89 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3333672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181Y]½ì’2À- _wandbruntime‚1990  _runtime‚1990L3ÿÁù 1loss/FBt2/assignment_loss‚0.08376091966045308 .loss/FBt2/detection_loss‚2.325196993297661 /loss/FRt1/assignment_loss‚3.327797339724207 .loss/FRt2/assignment_loss‚2.01492757148874 0loss/FBt1/assignment_loss‚0.3872395136086511 1loss/SRqqt1/detection_loss‚2.2150633658323935 1loss/SRqqt2/detection_loss‚0.7893368116101087 epoch‚3 _runtime‚1990.516042755 2loss/SRqqt2/assignment_loss‚0.8909170913813664 /loss/FRt1/detection_loss‚0.6388334557289651 .loss/FBt1/detection_loss‚1.938364670676147 trainer/global_step‚1099 2loss/SRqqt1/assignment_loss‚1.2823553949948927 /loss/FRt2/detection_loss‚1.5020334194869172 $loss/total_loss‚17.3958265474905 % _timestamp‚1.7848412271588523e+09 _step‚90B”ÈtÂý /loss/FRt1/assignment_loss‚3.327797339724207 .loss/FRt2/assignment_loss‚2.01492757148874 0loss/FBt1/assignment_loss‚0.3872395136086511 1loss/SRqqt1/detection_loss‚2.2150633658323935 1loss/SRqqt2/detection_loss‚0.7893368116101087 epoch‚3 _runtime‚1990.516042755 2loss/SRqqt2/assignment_loss‚0.8909170913813664 /loss/FRt1/detection_loss‚0.6388334557289651 .loss/FBt1/detection_loss‚1.938364670676147 trainer/global_step‚1099 2loss/SRqqt1/assignment_loss‚1.2823553949948927 /loss/FRt2/detection_loss‚1.5020334194869172 $loss/total_loss‚17.3958265474905 % _timestamp‚1.7848412271588523e+09 _step‚90 1loss/FBt2/assignment_loss‚0.08376091966045308 .loss/FBt2/detection_loss‚2.325196993297661Z¿Ý0ŸÃ‚(:” ˆŠÓ‡ÙÛ³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.245' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.08166666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\¶ç=Ä‚(:²  ˆŠÓÖ³ø³ network.recv‚163613 disk.drbd1584.in‚0 disk.rbd1.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0( proc.memory.availableMB‚ 5633.3671875* proc.memory.percent‚11.718734105428059 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 57.921875 disk.dm-0.in‚ 44.78515625% disk.nvme0n1.out‚5888.29833984375 proc.cpu.threads‚17 network.sent‚ 1.783498e+06% disk./.usageGB‚1260.8454933166504 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.drbd2111.out‚0" disk.dm-0.out‚5888.29833984375 disk.rbd1.out‚ 9.76171875 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0# memory_percent‚77.0777702331543 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 57.921875 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 44.78515625 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 cpu‚25.210539737488322 disk.drbd1931.out‚0) disk./.usagePercent‚70.51308045130409 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0–°Þ ŸÅ‚(:” žˆŠÓ¡Ôݳ gpu.0.gpu‚85.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚346.874' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚115.62466666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÇ AÆ‚(:¶  žˆŠÓÈ–ÿ³) proc.memory.availableMB‚ 5633.37890625 disk.rbd1.out‚ 9.76171875 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 58.3515625% disk.nvme0n1.out‚5888.54052734375 disk.drbd1550.out‚0 disk.drbd2118.in‚0 cpu‚25.210950333504115 disk.drbd1226.in‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.718734105428059 network.sent‚ 1.783564e+06 disk.nvme0n1.in‚ 44.78515625 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0$ memory_percent‚77.07772254943848 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.sda.out‚ 58.3515625 disk.drbd1673.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625 disk.sda2.in‚0.015625) disk./.usagePercent‚70.51307725123647 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 network.recv‚163679 disk.drbd1550.in‚0" disk.dm-0.out‚5888.54052734375 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1938.out‚0 disk.dm-0.in‚ 44.78515625 disk.drbd1285.out‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8454360961914 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0),¨ÜžÇ‚(:“ ­ˆŠÓˆ³³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚283.194' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.39800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ÍÉ6È‚(:«  ­ˆŠÓ­îÒ³ network.recv‚163787 disk.drbd1673.out‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.718734105428059 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 5633.37890625 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0$ memory_percent‚77.07772254943848 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 45.0859375 disk.sda1.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚58.9375 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 cpu‚25.211365185246 disk.sda.out‚58.9375 disk.drbd1828.out‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51306701102011! disk.dm-0.out‚5889.0009765625 disk.drbd2078.in‚0% disk./.usageGB‚1260.8452529907227 disk.drbd2111.out‚0 disk.rbd1.out‚ 9.7939453125$ disk.nvme0n1.out‚5889.0009765625 network.sent‚ 1.783672e+06 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625 disk.dm-0.in‚ 45.0859375# proc.memory.rssMB‚ 2879.99609375 proc.cpu.threads‚17O³Ã 7É‚(:¬  ¼ˆŠÓÇÂ÷³ disk.drbd1289.out‚0 disk.dm-0.in‚ 45.0859375 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0% disk./.usageGB‚1260.8452339172363 disk.drbd1673.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.015625 disk.sda2.out‚ 59.1796875 disk.drbd1828.out‚0 disk.drbd1510.in‚0% proc.memory.availableMB‚ 5633.1875 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 network.recv‚166705 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 9.7939453125 disk.sda1.out‚0# proc.memory.rssMB‚ 2879.99609375* proc.memory.percent‚11.718734105428059 network.sent‚ 1.845949e+06 disk.drbd1828.in‚0$ memory_percent‚77.07850138346353 disk.sda2.in‚0.015625 disk.drbd1356.out‚0# disk.nvme0n1.out‚5889.548828125 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚70.5130659443309 disk.sda.out‚ 59.1796875 disk.dm-0.out‚5889.548828125 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 45.0859375 cpu‚25.22839546334502 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0“¢›“Ê‚(:ˆ ¼ˆŠÓƒš×³ gpu.0.gpu‚95.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.869' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.623 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ—‡è2Ë- _wandbruntime‚2044  _runtime‚20449ªzmÚ;ÌÔw 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07384826987981796 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09412354230880737 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4737631380558014 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00501212989911437 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 trainer/global_step‚1119 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6113392114639282 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7438566088676453 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2803991138935089 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9675980806350708 *jet/accuracy_2_of_4‚0.6137309670448303 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.21597908437252045 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5813532471656799 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.1785714328289032 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4923076927661896 *jet/accuracy_1_of_2‚0.6794935464859009 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4684002697467804 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9789233803749084 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9824819564819336 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9239979982376099 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.1451047658920288 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9285714030265808 particle/accuracy_1_of_2‚1 -"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8087218403816223 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.988095223903656 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08589743822813034 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9941685795783997 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4633101522922516 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.14761146903038025 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1043819859623909 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9619989395141602 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06196081265807152 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.45365914702415466 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44821032881736755 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026281874626874924 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_2_of_3‚0.3404114544391632 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02102980948984623 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.41312095522880554 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10866905003786087 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4075496792793274 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5565560460090637 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4226888120174408 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9924836754798889 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996472179889679 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9967532157897949 epoch‚3 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5194724202156067 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43967390060424805 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05471637099981308 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4777557849884033 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9917130470275879 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_3‚0.03829488158226013 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9842001795768738 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.40668004751205444 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.41568025946617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4159667491912842 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0760113000869751 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1666666716337204 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.993868887424469 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9606395363807678 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.43665459752082825 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9992063641548157 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09794231504201889 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5141056776046753 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003418597159907222 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_2_of_2‚0.09309697151184082 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16812951862812042 /particle/accuracy_4_of_4‚0.9600061774253845 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2815729081630707 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2407415211200714 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06259845942258835 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5581961274147034 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2317773401737213 _runtime‚2044.721850197 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1510539948940277 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9821428656578064 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4377289116382599 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1785714328289032 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1575317680835724 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 &particle/f_score‚0.590772271156311 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946145415306091 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18730182945728302 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.837302565574646 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9972527623176575 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8342623710632324 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2729587256908417 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.39035463333129883 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5441077351570129 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24042390286922455 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.44193214178085327 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2317773401737213 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5053750276565552 +particle/sensitivity‚0.5861518979072571 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18933983147144318 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9845804572105408 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17703919112682343 +jet/accuracy_3_of_4‚0.05072423070669174 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06319058686494827 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48162218928337097 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4171936810016632 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20014457404613495 _step‚91 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9924836754798889 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6527943015098572 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6858199238777161 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3863636255264282 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.547640860080719 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4750765860080719 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9615477919578552 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4369190037250519 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.38795968890190125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07241647690534592 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14182351529598236 jet/accuracy_4_of_4‚0 +particle/specificity‚0.8912416100502014 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.985981822013855 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9964285492897034 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7validation_average_jet_accuracy‚0.26506099104881287 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_1_of_3‚1 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.060130633413791656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4848484694957733 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06421049684286118 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9680941700935364 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05059663578867912 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48162218928337097 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4834131896495819 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06259845942258835 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2500036060810089 *jet/accuracy_1_of_3‚0.8273068070411682 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17115400731563568 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4040893614292145 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.784841281363758e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5057447552680969 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.058945897966623306 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5256504416465759 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001849553664214909 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4393939673900604 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4848484694957733 *loss/val_total_loss‚17.279197567458503 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5010701417922974 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9823235273361206 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12830327451229095 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18066203594207764 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43476277589797974 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.35712406039237976 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9953917264938354 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4071815013885498 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07363035529851913 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13573209941387177 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24950222671031952 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8257765769958496 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932451248168945 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.47301700711250305 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9625397324562073 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4604012370109558 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3874928951263428 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5053750276565552 *jet/accuracy_1_of_4‚0.8833770751953125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4226742386817932 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3879595398902893 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.39380794763565063 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5767228007316589 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009992496110498905 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918552041053772 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3146006762981415 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9818575978279114 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6023440957069397Nân ÍØw 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4393939673900604 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4848484694957733 *loss/val_total_loss‚17.279197567458503 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5010701417922974 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9823235273361206 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12830327451229095 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18066203594207764 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43476277589797974 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.35712406039237976 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9953917264938354 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4071815013885498 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07363035529851913 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13573209941387177 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24950222671031952 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8257765769958496 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932451248168945 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.47301700711250305 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9625397324562073 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4604012370109558 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3874928951263428 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5053750276565552 *jet/accuracy_1_of_4‚0.8833770751953125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4226742386817932 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3879595398902893 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.39380794763565063 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5767228007316589 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009992496110498905 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918552041053772 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3146006762981415 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9818575978279114 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6023440957069397 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07384826987981796 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09412354230880737 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4737631380558014 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00501212989911437 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 trainer/global_step‚1119 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6113392114639282 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7438566088676ú–>Žp/453 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2803991138935089 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9675980806350708 *jet/accuracy_2_of_4‚0.6137309670448303 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.21597908437252045 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5813532471656799 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.1785714328289032 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4923076927661896 *jet/accuracy_1_of_2‚0.6794935464859009 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4684002697467804 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9789233803749084 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9824819564819336 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9239979982376099 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.1451047658920288 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9285714030265808 particle/accuracy_1_of_2‚1 -"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8087218403816223 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.988095223903656 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08589743822813034 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9941685795783997 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4633101522922516 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.14761146903038025 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1043819859623909 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9619989395141602 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06196081265807152 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.45365914702415466 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44821032881736755 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026281874626874924 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_2_of_3‚0.3404114544391632 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02102980948984623 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.41312095522880554 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10866905003786087 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4075496792793274 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5565560460090637 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4226888120174408 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9924836754798889 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996472179889679 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9967532157897949 epoch‚3 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5194724202156067 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43967390060424805 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05471637099981308 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4777557849884033 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9917130470275879 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_3‚0.03829488158226013 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9842001795768738 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.40668004751205444 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.41568025946617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4159667491912842 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0760113000869751 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1666666716337204 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.993868887424469 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9606395363807678 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.43665459752082825 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9992063641548157 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09794231504201889 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5141056776046753 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003418597159907222 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_2_of_2‚0.09309697151184082 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16812951862812042 /particle/accuracy_4_of_4‚0.9600061774253845 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2815729081630707 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2407415211200714 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06259845942258835 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5581961274147034 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2317773401737213 _runtime‚2044.721850197 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1510539948940277 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9821428656578064 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4377289116382599 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1785714328289032 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1575317680835724 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 &particle/f_score‚0.590772271156311 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946145415306091 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18730182945728302 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.837302565574646 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9972527623176575 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8342623710632324 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2729587256908417 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.39035463333129883 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5441077351570129 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24042390286922455 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.44193214178085327 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2317773401737213 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5053750276565552 +particle/sensitivity‚0.5861518979072571 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18933983147144318 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9845804572105408 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17703919112682343 +jet/accuracy_3_of_4‚0.05072423070669174 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06319058686494827 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48162218928337097 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4171936810016632 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20014457404613495 _step‚91 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9924836754798889 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6527943015098572 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6858199238777161 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3863636255264282 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.547640860080719 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4750765860080719 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9615477919578552 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4369190037250519 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.38795968890190125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07241647690534592 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14182351529598236 jet/accuracy_4_of_4‚0 +particle/specificity‚0.8912416100502014 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.985981822013855 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9964285492897034 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7validation_average_jet_accuracy‚0.26506099104881287 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_1_of_3‚1 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.060130633413791656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4848484694957733 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06421049684286118 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9680941700935364 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05059663578867912 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48162218928337097 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4834131896495819 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06259845942258835 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2500036060810089 *jet/accuracy_1_of_3‚0.8273068070411682 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17115400731563568 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4040893614292145 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.784841281363758e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5057447552680969 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.058945897966623306 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5256504416465759 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001849553664214909 ("Purity/2FRt2SRqqt*FBt/event_purity‚0[Ô§ÑIŸÎ‚(:” ˈŠÓˆœÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚177.362' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚59.120666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ž,AÏ‚(:¶  ˈŠÓÓâø³ disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0% disk.nvme0n1.out‚6859.98095703125 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1828.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 45.0859375% disk./.usageGB‚1260.8490371704102$ memory_percent‚75.49473444620767 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 9.9501953125 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0 network.recv‚166771 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 45.0859375 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.sda.out‚ 59.60546875 disk.sda2.out‚ 59.60546875 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 cpu‚25.22902919150462 network.sent‚ 1.846015e+06 disk.sda.in‚0.015625 disk.drbd2111.in‚0( proc.memory.availableMB‚ 6022.4140625# proc.memory.rssMB‚ 2879.99609375 disk.sda1.out‚0 disk.sda2.in‚0.015625* proc.memory.percent‚11.718734105428059) disk./.usagePercent‚70.51327864215823 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0" disk.dm-0.out‚6859.98095703125q ¡*žÐ‚(:“ ÚˆŠÓÅŸã³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚282.884' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.29466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¥Rd;Ñ‚(:°  ÚˆŠÓ‡Èÿ³ disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 45.0859375 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 45.0859375 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0% proc.memory.availableMB‚ 6324.1875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2879.99609375 network.recv‚168177 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.718734105428059% disk./.usageGB‚1260.8450050354004$ memory_percent‚74.26681518554688 disk.drbd1356.out‚0! disk.dm-0.out‚7156.5751953125 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.rbd1.out‚ 9.9501953125 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd1817.out‚0 cpu‚25.229415047578378) disk./.usagePercent‚70.51305314406045 disk.drbd2111.out‚0 disk.drbd1226.in‚0 network.sent‚ 1.854218e+06 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.sda.in‚0.015625 disk.sda.out‚ 60.1171875$ disk.nvme0n1.out‚7156.5751953125 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.015625 disk.drbd1550.in‚0 disk.sda2.out‚ 60.1171875ÿøå4Ò‚(:©  鈊ÓÐëϳ disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.015625 network.recv‚168285 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚60.6875 cpu‚25.22980607214699 disk.drbd1356.out‚0! disk.dm-0.out‚7831.0322265625* proc.memory.percent‚11.718734105428059 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0$ disk.nvme0n1.out‚7831.0322265625 disk.drbd2118.out‚0$ disk./.usageGB‚1260.844985961914 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.out‚60.6875 disk.drbd1938.in‚0 network.sent‚ 1.854326e+06 disk.drbd2111.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51305207737126 disk.drbd1356.in‚0 disk.rbd1.out‚ 9.982421875 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0# memory_percent‚75.5061149597168 disk.nvme0n1.in‚ 45.0859375 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1552.in‚0 disk.dm-0.in‚ 45.0859375( proc.memory.availableMB‚ 6019.6171875 disk.drbd1673.in‚0 disk.sda2.in‚0.015625#øjŒžÓ‚(:“ 鈊Ó’ gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚285.068' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.02266666666667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 –ŸV;Ô‚(:°  øˆŠÓãÃͳ$ disk.nvme0n1.out‚7831.2119140625 network.recv‚168351 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.sda2.in‚0.015625 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0( proc.memory.availableMB‚ 5742.7890625 cpu‚25.22994853142978 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 9.982421875# proc.memory.rssMB‚ 2879.99609375$ disk./.usageGB‚1260.844985961914 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51305207737126 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 45.0859375 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.sda2.out‚ 60.9765625* proc.memory.percent‚11.718734105428059 disk.sda1.in‚0 disk.sda.out‚ 60.9765625! disk.dm-0.out‚7831.2119140625 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 45.0859375$ memory_percent‚76.63253148396811 network.sent‚ 1.854392e+06ŵ~žÕ‚(:“ øˆŠÓ§¢®³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚279.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.00033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_‹'2Ö- _wandbruntime‚2100  _runtime‚2100òÆèÇ×Á ' lr-AdamW/pg1‚0.00011597343730509454 ' lr-AdamW/pg2‚0.00011597343730509454 trainer/global_step‚1149 $ _timestamp‚1.784841337027196e+09 _runtime‚2100.384842395 _step‚92Y1ËØÅ ' lr-AdamW/pg2‚0.00011597343730509454 trainer/global_step‚1149 $ _timestamp‚1.784841337027196e+09 _runtime‚2100.384842395 _step‚92 ' lr-AdamW/pg1‚0.00011597343730509454\çóœ2Ù- _wandbruntime‚2100  _runtime‚2100(Ï‹ç'ÚáO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10682841 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330944812169168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332143450418346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332277097841152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1469304803328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331076448933888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9984192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10682841 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1197268209924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1199277297664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10672777 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 _runtime‚ 2100.38552697 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10672777 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6575781 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848413370280564e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332275726231552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚332 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1200622733824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚1149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331076475107328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90761514023936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚93 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332275726231552 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92231186918400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9984186 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9984192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3507837 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1469259827712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20667033 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ÅG]æÛåO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331076448933888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3068276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9984186 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1199277297664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10672777 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 _runtime‚ 2100.38552697 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10672777 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 E;DeviceStatsMonitor.on_train_batch_start/allocation.alö+á!l.peak‚12299 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6575781 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848413370280564e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332275726231552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚332 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1200622733824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚1149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331076475107328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90761514023936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚93 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332275726231552 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92231186918400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9984186 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9984192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3507837 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1469259827712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20667033 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10682841 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330944812169168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332143450418346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332277097841152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1469304803328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331076448933888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9984192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10682841 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1197268209924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6581406 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3510816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331355358224384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848413385004902e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331355384397824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9992553 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3070916 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20684496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92310611339776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10681859 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10681859 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20684496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90840100389888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331355384397824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332423272993874 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10691937 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332557029646848 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9992553 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚331223613942944 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1198289011564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331355358224384 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _step‚94 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328àC§ö&ÞðM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332557029646848 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9992553 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚331223613942944 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1198289011564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331355358224384 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _step‚94 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1200299805696 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6581406 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3510816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331355358224384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848413385004902e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331355384397824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9992553 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3070916 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20684496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92310611339776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10681859 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10681859 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20684496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90840100389888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331355384397824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332423272993874 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10691937 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Pë‚(:³  ÉŠÓ¥ºÿ³ disk.sda.out‚ 63.01171875 disk.drbd2078.in‚0% disk./.usageGB‚1260.8470077514648 disk.drbd2041.in‚0$ memory_percent‚81.03446960449219 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 45.0859375& proc.memory.availableMB‚ 4660.96875 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.718734105428059 disk.nvme0n1.in‚ 45.0859375 disk.drbd1550.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2879.99609375 cpu‚25.23186521683124 network.recv‚173503( disk./.usagePercent‚70.5131651464269 disk.drbd1226.in‚0 disk.drbd1673.out‚0" disk.dm-0.out‚7833.26220703125 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.015625 disk.sda2.in‚0.015625 disk.sda2.out‚ 63.01171875 proc.cpu.threads‚17 disk.drbd2111.out‚0% disk.nvme0n1.out‚7833.26220703125 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 10.4443359375 disk.drbd1938.in‚0 disk.drbd2078.out‚0 network.sent‚ 1.97938e+06ó¸£2ì- _wandbruntime‚2187  _runtime‚2187§÷¿Èí ' lr-AdamW/pg2‚0.00011583713796258845 trainer/global_step‚1199 % _timestamp‚1.7848414239786375e+09 _runtime‚2187.336889422 _step‚96 ' lr-AdamW/pg1‚0.00011583713796258845¿w˜ÌîÆ % _timestamp‚1.7848414239786375e+09 _runtime‚2187.336889422 _step‚96 ' lr-AdamW/pg1‚0.00011583713796258845 ' lr-AdamW/pg2‚0.00011583713796258845 trainer/global_step‚1199`‡ÏŽé2ï- _wandbruntime‚2187  _runtime‚2187`ksÁì'ðæO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346272380312576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6856264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345021913458688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚1199 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346134642989954 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1249716704018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10402536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11138793 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1248372092732 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1251812290048 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10402542 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346272380312576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚366 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94691200412672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11138793 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1250466853888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3200271 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96223064873984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11128729 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚345021939632128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1250466853888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346273751922176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21541335 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531802708480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚371 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3655988 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96222635030016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚97 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345021913458688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 Ÿúð'ñêO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1248372092732 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1251812290048 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10402542 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346272380312576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚366 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94691200412672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11138793 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1250466853888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3200271 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96223064873984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11128729 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚345021939632128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1250466853888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346273751922176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21541335 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531802708480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚371 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3655988 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96222635030016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚97 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345021913458688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346272380312576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6856264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345021913458688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚1199 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346134642989954 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1249716704018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10402536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11138793 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224aÈŠ]ã2ò- _wandbruntime‚2188  _runtime‚2188ã÷=µóðM KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345300848922624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6861822 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21548714 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96302895663104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10410903 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345300848922624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1251489361920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1251489361920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11137811 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1252834805248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1533051333120 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10410909 NADeviceStatsMonitoܼîôAr.on_train_batch_end/allocation.small_pool.freed‚11137811 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345163702631744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21558798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346414465565482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21558798 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3202911 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3658546 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10410903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94769418687488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6861457 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345300822749184 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96302470020608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1199 _step‚98 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11147889 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1533108884480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚345300822749184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21548714 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005Æ/+ú&ôôM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10410903 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345300848922624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1251489361920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1251489361920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11137811 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1252834805248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1533051333120 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10410909 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11137811 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345163702631744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21558798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346414465565482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21558798 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3202911 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3658546 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10410903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94769418687488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6861457 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345300822749184 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96302470020608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1199 _step‚98 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11147889 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1533108884480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚345300822749184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21548714 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345300848922624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6861822 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21548714 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96302895663104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864b3K7ù2õ- _wandbruntime‚2188  _runtime‚2188•ûÅMöý 0loss/FRt2/assignment_loss‚1.8661263621924968 0loss/FBt1/assignment_loss‚0.3820150032124454 1loss/FBt2/assignment_loss‚0.09626836673561834 .loss/FRt1/detection_loss‚0.671959300915398 /loss/FRt2/detection_loss‚1.4738492088208108 _step‚99 1loss/SRqqt1/detection_loss‚1.7954950435257657 1loss/SRqqt2/detection_loss‚0.7958117180363715 /loss/FBt1/detection_loss‚1.4360974219739684 &loss/total_loss‚14.907073413823714 trainer/global_step‚1199 % _timestamp‚1.7848414254957016e+09 1loss/SRqqt1/assignment_loss‚1.166900647142322 2loss/SRqqt2/assignment_loss‚0.8642564968307833 epoch‚4 /loss/FRt1/assignment_loss‚3.171573439213239 /loss/FBt2/detection_loss‚1.1867204052244926 _runtime‚2188.852959676¡¹Þ÷ 1loss/SRqqt1/detection_loss‚1.7954950435257657 1loss/SRqqt2/detection_loss‚0.7958117180363715 /loss/FBt1/detection_loss‚1.4360974219739684 &loss/total_loss‚14.907073413823714 trainer/global_step‚1199 % _timestamp‚1.7848414254957016e+09 1loss/SRqqt1/assignment_loss‚1.166900647142322 2loss/SRqqt2/assignment_loss‚0.8642564968307833 epoch‚4 /loss/FRt1/assignment_loss‚3.171573439213239 /loss/FBt2/detection_loss‚1.1867204052244926 _runtime‚2188.852959676 0loss/FRt2/assignment_loss‚1.8661263621924968 0loss/FBt1/assignment_loss‚0.3820150032124454 1loss/FBt2/assignment_loss‚0.09626836673561834 .loss/FRt1/detection_loss‚0.671959300915398 /loss/FRt2/detection_loss‚1.4738492088208108 _step‚99cv,*ø‚(:„ Ò‰ŠÓü’ܳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.9' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.3 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê&zvCù‚(:¸  Ò‰ŠÓâ¨ü³ disk.drbd1552.out‚0 disk.sda2.in‚0.015625 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0$ memory_percent‚81.03445370992026) proc.memory.availableMB‚ 4660.97265625# proc.memory.rssMB‚ 2879.99609375 disk.dm-0.in‚ 46.79296875( disk./.usagePercent‚70.5131651464269 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 46.79296875 disk.drbd1673.out‚0 disk.sda2.out‚ 63.5546875 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0% disk.nvme0n1.out‚7833.40283203125 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 network.recv‚173611 disk.drbd1938.in‚0 disk.rbd1.in‚0 network.sent‚ 1.979488e+06 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.015625 disk.drbd1931.out‚0 disk.drbd1289.in‚0" disk.dm-0.out‚7833.40283203125 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.718734105428059 disk.drbd1289.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8470077514648 disk.sda.out‚ 63.5546875 disk.rbd1.out‚ 10.5068359375 cpu‚25.232100507377773É:žú‚(:“ ቊÓŠ¸¶³ gpu.0.gpu‚74.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.487' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.82900000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾5¢Œ@û‚(:µ  ቊÓ·„Ó³ proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 47.01953125 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 63.921875 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.dm-0.in‚ 47.01953125$ memory_percent‚81.03470802307129# proc.memory.rssMB‚ 2879.99609375 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.5068359375 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 63.921875 disk.drbd1673.out‚0 network.sent‚ 1.985007e+06 disk.sda1.in‚0 disk.sda.in‚0.015625 disk.drbd1289.out‚0 cpu‚25.23256313160359 disk.drbd1552.in‚0) proc.memory.availableMB‚ 4660.91015625 disk.drbd2111.out‚0% disk.nvme0n1.out‚7833.72216796875" disk.dm-0.out‚7833.72216796875* proc.memory.percent‚11.718734105428059( disk./.usagePercent‚70.5131651464269 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 network.recv‚174611% disk./.usageGB‚1260.8470077514648 disk.drbd1226.in‚0 disk.sda2.in‚0.015625 disk.drbd1938.out‚0‚vsžü‚(:“ ð‰ŠÓ™ÜÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.254' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.75133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äqùÍCý‚(:¸  ð‰ŠÓÆúó³ disk.drbd1552.out‚0" disk.dm-0.out‚7833.89404296875 disk.drbd1828.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚7833.89404296875 network.recv‚174677 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 64.15234375 disk.drbd2041.out‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51316642645395 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 47.01953125 disk.drbd1510.in‚0% disk./.usageGB‚1260.8470306396484 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0$ memory_percent‚81.03505770365396 disk.drbd1938.in‚0* proc.memory.percent‚11.718734105428059 disk.drbd2118.in‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 4660.82421875 cpu‚25.232673139106986 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda.out‚ 64.15234375 disk.rbd1.in‚0 disk.rbd1.out‚ 10.5390625 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1931.in‚0 disk.drbd2078.in‚0 network.sent‚ 1.985073e+06 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 47.01953125 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0NªVc“þ‚(:ˆ ÿ‰ŠÓÇß±³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚283.209' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.403 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êž @ÿ‚(:µ  ÿ‰ŠÓŒÆг disk.drbd2078.out‚0 disk.sda2.in‚0.015625 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.718734105428059) disk./.usagePercent‚70.51316535976474 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 10.6328125 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0$ disk.nvme0n1.out‚7834.1982421875 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 64.55078125 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0$ memory_percent‚81.03413581848145 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 network.recv‚174785 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 network.sent‚ 1.985181e+06 disk.drbd1931.in‚0! disk.dm-0.out‚7834.1982421875 disk.drbd1584.out‚0 disk.sda2.out‚ 64.55078125 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 47.01953125 disk.sda1.out‚0) proc.memory.availableMB‚ 4661.05078125# proc.memory.rssMB‚ 2879.99609375$ disk./.usageGB‚1260.847011566162 disk.drbd1965.in‚0 cpu‚25.233014527400083 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 47.01953125 disk.drbd2111.in‚0 disk.sda.in‚0.015625 disk.drbd1938.out‚0Jq ™”€ ‚(:‰ ŽŠŠÓǀɳ gpu.0.gpu‚63.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.314' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.438 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žò¢7 ‚(:¬  ŽŠŠÓ‰–î³ disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51316535976474 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 10.6953125 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0! disk.dm-0.out‚7834.3974609375 disk.drbd1356.out‚0$ disk./.usageGB‚1260.847011566162 disk.drbd1965.out‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.718734105428059 disk.sda2.out‚64.84375 disk.drbd1931.out‚0 disk.dm-0.in‚ 47.01953125 disk.nvme0n1.in‚ 47.01953125 cpu‚25.233339738749972 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 network.sent‚ 2.050397e+06 disk.sda.in‚0.015625 disk.sda.out‚64.84375 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.sda1.out‚0$ memory_percent‚81.03523254394531& proc.memory.availableMB‚ 4660.78125 disk.drbd1673.in‚0 network.recv‚177179 disk.drbd1817.out‚0 disk.sda2.in‚0.015625$ disk.nvme0n1.out‚7834.3974609375 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1931.in‚0’6É®=‚ ‚(:²  ŠŠÓìÝà³ disk.drbd2111.in‚0% disk.nvme0n1.out‚7834.66650390625' proc.memory.availableMB‚ 4660.671875 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.sda.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 47.01953125# proc.memory.rssMB‚ 2879.99609375% disk./.usageGB‚1260.8469924926758 disk.drbd1828.out‚0$ memory_percent‚81.03567759195963 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.015625 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 network.sent‚ 2.058605e+06 disk.rbd1.out‚ 10.6953125 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 47.01953125 disk.sda.out‚ 65.140625 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.718734105428059 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51316429307553 disk.drbd1285.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 cpu‚25.233450827950186 network.recv‚178433 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0" disk.dm-0.out‚7834.66650390625 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 65.140625 disk.drbd1931.in‚0 proc.cpu.threads‚17±=£«žƒ ‚(:“ ŠŠÓÿœ¿³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚274.463' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.48766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05`02„ - _wandbruntime‚2274  _runtime‚2274OÞÖŠÇ… Á % _timestamp‚1.7848415107200174e+09 _runtime‚2274.077263909 _step‚100 & lr-AdamW/pg1‚0.0001156933314909366 & lr-AdamW/pg2‚0.0001156933314909366 trainer/global_step‚1249o úùˆ Å & lr-AdamW/pg1‚0.0001156933314909366 & lr-AdamW/pg2‚0.0001156933314909366 trainer/global_step‚1249 % _timestamp‚1.7848415107200174e+09 _runtime‚2274.077263909 _step‚100dA©V2‡ - _wandbruntime‚2274  _runtime‚2274 B¾ï ˆ çO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1303001846272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10820892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360269034393600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11594745 PCDevìøþiceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11584681 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3332276 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358824989546768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1300820586826 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360125835561562 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3332271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98620518710272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1594082392576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360269034393600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358967404156928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3803566 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 $ _timestamp‚1.784841510721046e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3803189 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98620150619136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1301656410112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11584681 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360270406003200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10820892 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7135842 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360270406003200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22415637 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1301656410112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10820886 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100214654467072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100214233011712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358967377983488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358825014974736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚1249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10820886 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358967377983488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1303001846272 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360124465522308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1594135756800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181qœ©Oñ'‰ ëO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10820892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360269034393600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11594745 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11584681 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3332276 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358824989546768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1300820586826 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360125835561562 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3332271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98620518710272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1594082392576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360269034393600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358967404156928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3803566 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 $ _timestamp‚1.784841510721046e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3803189 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98620150619136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1301656410112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11584681 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360270406003200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10820892 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7135842 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360270406003200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22415637 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1301656410112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10820886 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100214654467072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100214233011712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚622 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358967377983488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358825014974736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚1249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10820886 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358967377983488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1303001846272 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360124465522308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1594135756800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1303001846272eè…:2Š - _wandbruntime‚2275  _runtime‚2275Þ…Aéù&‹ óM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359246313447424 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7141063 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10829253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 % _timestamp‚1.7848415121980922e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3334916 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10829253 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11593763 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3806528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359246287273984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11603841 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360550337808896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359246287273984 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22433100 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100294484687872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100294063239680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1302678918144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98699105076224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22423016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11603841 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360404288097724 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚102 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚1249 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360550337808896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1302678918144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3334911 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1301841388578 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10829259 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359103791320544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359246313447424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1304024361472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7141444 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1595379611648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22433100 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22423016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1304024361472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚2275.555678227 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98698736985088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1300496777180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824ã¦ÃŒ ÷M M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10829259 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359103791320544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359246313447424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1304024361472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7141444 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1595379611648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚654 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22433100 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22423016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1304024361472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚2275.555678227 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98698736985088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1300496777180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359246313447424 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7141063 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10829253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 % _timestamp‚1.7848415121980922e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3334916 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10829253 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11593763 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3806528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359246287273984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11603841 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsDÙ:#:Monitor.on_train_batch_end/segment.small_pool.peak‚824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360550337808896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359246287273984 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22433100 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100294484687872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚622 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100294063239680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1302678918144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98699105076224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22423016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11603841 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360404288097724 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚102 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚1249 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360550337808896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1302678918144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3334911 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1301841388578 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 EBÃü2œ - _wandbruntime‚2360  _runtime‚2360O˜kÇ Á & lr-AdamW/pg2‚0.0001155420367398726 trainer/global_step‚1299 % _timestamp‚1.7848415975989714e+09 _runtime‚2360.956190611 _step‚104 & lr-AdamW/pg1‚0.0001155420367398726eî™Ëž Å & lr-AdamW/pg1‚0.0001155420367398726 & lr-AdamW/pg2‚0.0001155420367398726 trainer/global_step‚1299 % _timestamp‚1.7848415975989714e+09 _runtime‚2360.956190611 _step‚104hJ5Z¶2Ÿ - _wandbruntime‚2360  _runtime‚2360ŠÝ3„ñ'  ëO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7416467 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1656289399296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102549468916736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374265688474624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1656349054976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23279869 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1354191402496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚399 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11239242 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374267060084224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12050697 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7416063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372912842508288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102549837007872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374265688474624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1350579858348 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464276 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372912842508288 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚2360.960924055 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3952191 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1354191402496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374267060084224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374115658093916 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11239242 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372912868681728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3951792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23289939 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104205758316032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11239236 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚404 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1352845966336 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374265688474624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1656349054976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23279869 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1354191402496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚399 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11239242 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374267060084224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12050697 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7416063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚654 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_sŒ3=-tart/allocated_bytes.large_pool.freed‚372912842508288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102549837007872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374265688474624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1350579858348 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464276 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372912842508288 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚2360.960924055 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3952191 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1354191402496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374267060084224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374115658093916 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11239242 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372912868681728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3951792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23289939 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104205758316032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11239236 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚404 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1352845966336 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104286009438208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚393 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373191751798784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3466916 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3466911 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23297318 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1353868474368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373043905437312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1352945271386 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23307402 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373191777972224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12049715 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12059793 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102628423373824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1355213917696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3955166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11247609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2362.439382269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12049715 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373191751798784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12059793 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3954773 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373043880009344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374545620273152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1657586064384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7422082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚398 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1657522221568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374396850708698 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _step‚106 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1355213917696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23307402 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7421684 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102628055282688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374395480669332 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373191777972224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1353868474368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299ç(’Âþ&¤ øM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12059793 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102628423373824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1355213917696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3955166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11247609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2362.439382269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12049715 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373191751798784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12059793 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3954773 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373043880009344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374545620273152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1657586064384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7422082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚398 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1657522221568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374396850708698 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _step‚106 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1355213917696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23307402 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7421684 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102628055282688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374395480669332 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373191777972224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1353868474368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104285577504256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104286009438208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚393 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373191751798784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3466916 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3466911 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23297318 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1353868474368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373043905437312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1352945271386 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23307402 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373191777972224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12049715 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864jUu „2¥ - _wandbruntime‚2362  _runtime‚2362|%h¦  0loss/FRt2/assignment_loss‚1.8986458176221603 1loss/FBt1/assignment_loss‚0.20015533369553395 &loss/total_loss‚17.490116498022534 % _timestamp‚1.7848415991166995e+09 _runtime‚2362.473960973 0loss/FRt1/assignment_loss‚3.0694301250853826 /loss/FRt1/detection_loss‚0.6267273702733779 1loss/SRqqt1/detection_loss‚1.4161625655581718 1loss/SRqqt2/detection_loss‚0.7815436266645837 /loss/FBt2/detection_loss‚2.5189859625255897 trainer/global_step‚1299 2loss/SRqqt2/assignment_loss‚0.8243817337943651 1loss/FBt2/assignment_loss‚0.06181460807710883 .loss/FRt2/detection_loss‚1.419994855300015 epoch‚4 _step‚107 2loss/SRqqt1/assignment_loss‚1.1280123426044573 /loss/FBt1/detection_loss‚3.5442621568217874Ûåeç § … 0loss/FRt2/assignment_loss‚1.8986458176221603 1loss/FBt1/assignment_loss‚0.20015533369553395 &loss/total_loss‚17.490116498022534 % _timestamp‚1.7848415991166995e+09 _runtime‚2362.473960973 0loss/FRt1/assignment_loss‚3.0694301250853826 /loss/FRt1/detection_loss‚0.6267273702733779 1loss/SRqqt1/detection_loss‚1.4161625655581718 1loss/SRqqt2/detection_loss‚0.7815436266645837 /loss/FBt2/detection_loss‚2.5189859625255897 trainer/global_step‚1299 2loss/SRqqt2/assignment_loss‚0.8243817337943651 1loss/FBt2/assignment_loss‚0.06181460807710883 .loss/FRt2/detection_loss‚1.419994855300015 epoch‚4 _step‚107 2loss/SRqqt1/assignment_loss‚1.1280123426044573 /loss/FBt1/detection_loss‚3.5442621568217874k@?@¨ ‚(:µ  †‹ŠÓ¾Ê÷³ disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.drbd1965.out‚0 cpu‚25.23541800273298% disk./.usageGB‚1260.8470420837402 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 67.90234375 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0# memory_percent‚81.0309092203776 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 47.01953125% disk.nvme0n1.out‚7837.92431640625 disk.drbd1931.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 47.01953125* proc.memory.percent‚11.718734105428059 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.015625& proc.memory.availableMB‚ 4661.84375 network.recv‚183672 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd2041.out‚0 network.sent‚ 2.130256e+06) disk./.usagePercent‚70.51316706646747 disk.rbd1.in‚0 disk.sda2.out‚ 67.90234375 disk.rbd1.out‚ 11.072265625 disk.drbd1289.in‚0" disk.dm-0.out‚7837.92431640625 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0jkã© ‚(:’ †‹ŠÓͨӳ gpu.0.gpu‚62.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.36' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.45333333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TJØ´žª ‚(:“ •‹ŠÓÈÇà³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.664' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.55466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í#<« ‚(:±  •‹ŠÓ‘œ‚´ disk.dm-0.in‚ 47.23828125 disk.drbd2111.out‚0 disk.drbd1226.in‚0 network.recv‚183780 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2879.99609375 disk.sda.in‚0.015625 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0) disk./.usagePercent‚70.51309965170977 disk.rbd1.out‚ 11.072265625 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1673.in‚0 cpu‚25.23550664624676 disk.drbd1510.out‚0 disk.sda1.out‚0 network.sent‚ 2.130364e+06 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 68.30078125 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0# memory_percent‚81.0309092203776& proc.memory.availableMB‚ 4661.84375 disk.drbd1510.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8458366394043 disk.drbd1965.in‚0# disk.nvme0n1.out‚7838.595703125 disk.drbd2118.in‚0 disk.sda2.in‚0.015625 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.dm-0.out‚7838.595703125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.sda.out‚ 68.30078125* proc.memory.percent‚11.718734105428059y)tÎq¬ ‚(:ˆ ¤‹ŠÓð³³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055ä"* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.762' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.254 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ɪÝX>­ ‚(:³  ¤‹ŠÓ‹´Õ³ disk.drbd1931.out‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.8458366394043 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.dm-0.out‚7838.810546875 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 11.1044921875& proc.memory.availableMB‚ 4661.84375) disk./.usagePercent‚70.51309965170977 disk.sda2.out‚ 68.56640625 disk.drbd2041.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0* proc.memory.percent‚11.718734105428059 disk.drbd1965.in‚0# memory_percent‚81.0309092203776 cpu‚25.235911698696402 network.recv‚186548 disk.sda2.in‚0.015625 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1356.out‚0 disk.sda.out‚ 68.56640625 disk.drbd1673.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 47.23828125 network.sent‚ 2.185043e+06 disk.drbd1550.in‚0 disk.drbd1584.out‚0# disk.nvme0n1.out‚7838.810546875 disk.sda.in‚0.015625 disk.drbd1673.in‚0‰þ†ž® ‚(:“ ³‹ŠÓÖûº³ gpu.0.gpu‚66.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.127' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.37566666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ð¸=¯ ‚(:²  ³‹ŠÓ¥»Û³ disk.sda.in‚0.015625 disk.drbd1673.in‚0 disk.drbd2041.out‚0 network.recv‚186614# disk.nvme0n1.out‚7839.138671875 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0( disk./.usagePercent‚70.5133249364697 disk.drbd2118.in‚0* proc.memory.percent‚11.718734105428059 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8498649597168 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.sda2.out‚ 69.02734375 disk.nvme0n1.in‚ 47.23828125 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda.out‚ 69.02734375 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0$ memory_percent‚81.03103637695312 disk.rbd1.in‚0# proc.memory.rssMB‚ 2879.99609375 cpu‚25.236201164242004 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 47.23828125 disk.dm-0.out‚7839.138671875 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 11.1982421875 network.sent‚ 2.185109e+06 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0% proc.memory.availableMB‚ 4661.8125 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0%.pÌž° ‚(:“ ‹ŠÓæ°Õ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.509' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.16966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B»& <± ‚(:±  ‹ŠÓ–ô³# disk.nvme0n1.out‚7839.326171875 disk.drbd1552.in‚0) disk./.usagePercent‚70.51332301642913% disk./.usageGB‚1260.8498306274414 disk.drbd1285.in‚0 disk.sda.in‚0.015625 disk.drbd1965.in‚0 disk.sda2.in‚0.015625 disk.sda.out‚ 69.3203125 disk.rbd1.out‚ 11.2607421875 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1817.in‚0 disk.dm-0.out‚7839.326171875 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0$ memory_percent‚81.03103637695312 disk.drbd1938.in‚0* proc.memory.percent‚11.718734105428059 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 47.23828125 disk.drbd2111.in‚0 cpu‚25.236281081368872 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2879.99609375 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 network.sent‚ 2.237575e+06 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 69.3203125 disk.drbd2118.in‚0 disk.drbd2041.in‚0 network.recv‚189067 disk.drbd1584.in‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1226.out‚0 disk.drbd1510.out‚0% proc.memory.availableMB‚ 4661.8125a—s£Ÿ² ‚(:” Ñ‹ŠÓñ¡Þ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚307.308' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.43599999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f$~B³ ‚(:·  Ñ‹ŠÓ¹Ãû³ disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0$ memory_percent‚81.03103637695312 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 47.23828125 disk.drbd2111.in‚0 disk.drbd1226.in‚0% proc.memory.availableMB‚ 4661.8125 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0" disk.dm-0.out‚7839.62060546875 disk.drbd2111.out‚0% disk.nvme0n1.out‚7839.62060546875 disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 cpu‚25.236564786459834 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 network.sent‚ 2.237641e+06% disk./.usageGB‚1260.8498306274414 disk.sda2.in‚0.015625 disk.sda2.out‚ 69.71484375 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 47.23828125# proc.memory.rssMB‚ 2879.99609375 disk.drbd1584.out‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.718734105428059 proc.cpu.threads‚17 disk.sda.in‚0.015625 disk.drbd2041.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51332301642913 disk.drbd1356.out‚0 disk.drbd1550.out‚0 network.recv‚189133 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1938.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 69.71484375 disk.drbd1817.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 11.2607421875!„N’2´ - _wandbruntime‚2448  _runtime‚2448 þyɵ à % _timestamp‚1.7848416846503677e+09 _runtime‚2448.007619608 _step‚108 ' lr-AdamW/pg1‚0.00011538327354067208 ' lr-AdamW/pg2‚0.00011538327354067208 trainer/global_step‚1349[ÏÈIͶ Ç _runtime‚2448.007619608 _step‚108 ' lr-AdamW/pg1‚0.00011538327354067208 ' lr-AdamW/pg2‚0.00011538327354067208 trainer/global_step‚1349 % _timestamp‚1.7848416846503677e+09lUó×Ø2· - _wandbruntime‚2448  _runtime‚2448iÕÁ¡ð'¸ êO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388262342555648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24164241 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1404035522560 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3596276 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388262342555648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101604 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1718745669632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11657586 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1405380958720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12506649 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚1349 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚370 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3596271 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _runtime‚2448.009041414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386858307033088 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24154171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1404035522560 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106478787214336 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12496585 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4101239 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12496585 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386858333206528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108197485811200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386705166924368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 $ _timestamp‚1.784841684651493e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7697510 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386705192352336 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386858307033088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚388106850665524 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388263714165248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚388108220704778N¬ä.ô'¹ îO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386705192352336 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386858307033088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚388106850665524 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388263714165248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚388108220704778 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388262342555648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24164241 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1404035522560 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3596276 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388262342555648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101604 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1718745669632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11657586 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1405380958720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12506649 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚1349 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚370 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3596271 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _runtime‚2448.009041414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386858307033088 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24154171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1404035522560 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106478787214336 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12496585 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4101239 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12496585 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386858333206528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108197485811200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386705166924368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 $ _timestamp‚1.784841684651493e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7697510m û´2º - _wandbruntime‚2449  _runtime‚2449 Ü’Š» õM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386983968698144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388543645970944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4104638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4104274 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387137242497024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404049154194 trainer/global_step‚1349 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12515745 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387137216323584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7703554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7848416861318285e+09 _runtime‚2449.489442009 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12505667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11665953 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24181704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11665959 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11665959 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388388043280306 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7703185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ñÿ&¼ ùM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387137216323584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _step‚110 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24181704 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108277328035328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12515745 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1406403473920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24171620 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1405058030592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3598911 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106557741671424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387137242497024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106557373580288 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7703185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386983968698144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388543645970944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4104638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4104274 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387137242497024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404049154194 trainer/global_step‚1349 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12515745 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387137216323584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7703554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7848416861318285e+09 _runtime‚2449.489442009 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12505667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11665953 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24181704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11665959 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11665959 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388388043280306 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_st4甇art/requested_bytes.all.freed‚402098043237132 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400803771557888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1455225078784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112189895678464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12075936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1455225078784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3728271 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4250513 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1454132235250 _step‚113 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402258996636672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12962601 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1456570514944 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402258996636672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12962601 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400803797731328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400803771557888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110408105511936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400645281041136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112189478417408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1781372905472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400803797731328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25038543 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12075936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110408473603072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3728276 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12952537 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400645255613168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0±XiÊõ'Ï ïO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12952537 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7978435 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12075942 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402099413276386 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4250164 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402098043237132 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400803771557888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1455225078784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112189895678464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12075936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1455225078784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3728271 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4250513 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1454132235250 _step‚113 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402258996636672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12962601 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1456570514944 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402258996636672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12962601 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400803797731328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400803771557888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110408105511936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400645281041136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112189478417408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1781372905472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400803797731328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25038543 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12075936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110408473603072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3728276 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12952537 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400645255613168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402260368246272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1456570514944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25028473 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402260368246272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1453808425604 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1457593030144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3730911 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1457593030144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12084303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12971697 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12961619 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7984347 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12084303 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401082680848384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402540300051968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12084309 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚114 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12084309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12971697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4253081 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1782627395584 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25056006 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25045922 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848417727549572e+09 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25045922 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401082707021824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1456247586816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402538928435200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400924057386944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400924082814912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401082707021824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7983992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1456247586816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1782680752640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110487059969024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4253431 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1455153037002 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1453808425604 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1457593030144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3730911 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1457593030144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12084303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12971697 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12961619 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7984347 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12084303 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401082680848384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402540300051968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12084309 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚114 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12084309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12971697 ZuöÉͺMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4253081 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1782627395584 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25056006 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25045922 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848417727549572e+09 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25045922 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401082707021824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1456247586816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402538928435200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400924057386944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400924082814912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401082707021824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7983992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1456247586816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1782680752640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110487059969024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4253431 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1455153037002 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7392429113388062 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10469140857458115 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6708135604858398 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9924722909927368 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5511510372161865 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.49372291564941406 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12130255252122879 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5580533146858215 *jet/accuracy_1_of_2‚0.7214089035987854 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6005987524986267 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.030522465705871582 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11335702985525131 trainer/global_step‚1399 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9885801076889038 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1515793800354004 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6060605645179749 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5786721110343933 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009210062213242054 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11700467020273209 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5451489686965942 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6617705225944519 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4580903947353363 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.36356121301651 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17091870307922363 'particle/f_score‚0.5945221781730652 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 epoch‚4 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22934412956237793 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9942176938056946 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6782194972038269 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3266999125480652 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.3928571343421936 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11938638240098953 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7301154732704163 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.11558788269758224 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09610409289598465 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *particle/sensitivity‚0.586283802986145 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7678698301315308 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.944444477558136 *jet/accuracy_1_of_4‚0.9464741349220276 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5850287675857544 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.683887779712677 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15993353724479675 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34090909361839294 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5933955311775208 *jet/accuracy_1_of_1‚0.2539796233177185 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0694444477558136 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5161901712417603 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8796935677528381 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9965655207633972 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4831336438655853 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.113216333091259 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9894274473190308 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8333333730697632 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.606765627861023 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10029184818267822 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6957405805587769 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14021500945091248 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.8974359035491943 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7159558534622192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19202573597431183 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5511654019355774 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.33771270513534546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.11273316293954849 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4166666865348816 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.044601697474718094 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9802117347717285 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13879276812076569 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 *jet/accuracy_1_of_3‚0.8791176080703735 (particle/accuracy‚0.8284330368041992 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4925667643547058 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35256412625312805 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9702522158622742 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_4‚0.9919642806053162 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882721304893494 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12515857815742493 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0694444477558136 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚ 2555.46890878 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4825811982154846 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3944590985774994 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9965655207633972 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6063946485519409 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4508667588233948 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9865620732307434 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10029184818267822 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3928571343421936 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9823075532913208 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 % _timestamp‚1.7848417921108503e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7244941592216492 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5421360731124878 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08589743822813034 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 particle/accuracy_2_of_3‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7339264154434204 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚26.955262178095847 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6135820150375366 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 7validation_average_jet_accuracy‚0.30262884497642517 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930378794670105 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.40070268511772156 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_4‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.170986607670784 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9816418886184692 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08782640844583511 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16232872009277344 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8333333730697632 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6060605645179749 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.8974359035491943 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02568301558494568 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1538461595773697 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9895378947257996 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13857145607471466 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5821351408958435 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917477965354919 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5580533146858215 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 _step‚116 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1028117686510086 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5151515603065491 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8477053642272949 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9891899228096008 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.008503401651978493 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9828258156776428 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6964285969734192 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18941564857959747 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.657195508480072 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8304692506790161 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45869967341423035 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9934397339820862 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07914271205663681 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.479022353887558 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11077553778886795 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45390990376472473 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6984224319458008 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.11305495351552963 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9236111640930176 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851261377334595 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4284725487232208 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5899925827980042 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.440582811832428 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10713041573762894 *jet/accuracy_2_of_4‚0.7599766850471497 /particle/accuracy_4_of_4‚0.9534499049186707 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6970933675765991 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01865123026072979 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11772318929433823 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11737661063671112 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6959956884384155 *jet/accuracy_2_of_3‚0.4364486336708069 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2782960534095764 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5324657559394836 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.47435900568962097 *particle/specificity‚0.894588053226471 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4882695972919464 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5302093625068665 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918726682662964 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1146015077829361 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6964285969734192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9785404801368713 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2730737030506134 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08697552978992462 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7392429113388062 +jet/accuracy_3_of_4‚0.09438169002532959 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10713041573762894 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3246743679046631 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2986352741718292 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09908856451511383 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966319799423218 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.519468367099762 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10313931852579117øìÛ¸Ü ˆx 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3928571343421936 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9823075532913208 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 % _timestamp‚1.7848417921108503e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7244941592216492 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5421360731124878 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08589743822813034 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 particle/accuracy_2_of_3‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7339264154434204 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚26.955262178095847 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6135820150375366 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 7validation_average_jet_accuracy‚0.30262884497642517 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930378794670105 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.40070268511772156 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_4‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.170986607670784 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9816418886184692 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08782640844583511 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16232872009277344 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8333333730697632 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6060605645179749 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.8974359035491943 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02568301558494568 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1538461595773697 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9895378947257996 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13857145607471466 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5821351408958435 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917477965354919 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5580533146858215 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 _step‚116 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1028117686510086 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5151515603065491 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8477053642272949 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9891899228096008 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.008503401651978493 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9828258156776428 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6964285969734192 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18941564857959747 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.657195508480072 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8304692506790161 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45869967341423035 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9934397339820862 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07914271205663681 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.479022353887558 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11077553778886795 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45390990376472473 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6984224319458008 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.11305495351552963 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9236111640930176 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851261377334595 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4284725487232208 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5899925827980042 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.440582811832428 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10713041573762894 *jet/accuracy_2_of_4‚0.7599766850471497 /particle/accuracy_4_of_4‚0.9534499049186707 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6970933675765991 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01865123026072979 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11772318929433823 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11737661063671112 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6959956884384155 *jet/accuracy_2_of_3‚0.4364486336708069 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2782960534095764 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5324657559394836 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.47435900568962097 *particle/specificity‚0.894588053226471 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4882695972919464 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5302093625068665 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918726682662964 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1146015077829361 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6964285969734192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9785404801368713 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2730737030506134 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08697552978992462 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7392429113388062 +jet/accuracy_3_of_4‚0.09438169002532959 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10713041573762894 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3246743679046631 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2986352741718292 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09908856451511383 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966319799423218 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.519468367099762 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10313931852579117 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936150312423706 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9925594925880432 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8244773745536804 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5478100776672363 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt1SRqqt*FBt/FBt_pur~®êßVity‚0.9904761910438538 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7392429113388062 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10469140857458115 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6708135604858398 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9924722909927368 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5511510372161865 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.49372291564941406 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12130255252122879 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5580533146858215 *jet/accuracy_1_of_2‚0.7214089035987854 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6005987524986267 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.030522465705871582 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11335702985525131 trainer/global_step‚1399 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9885801076889038 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1515793800354004 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6060605645179749 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5786721110343933 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009210062213242054 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11700467020273209 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5451489686965942 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6617705225944519 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4580903947353363 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.36356121301651 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17091870307922363 'particle/f_score‚0.5945221781730652 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 epoch‚4 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22934412956237793 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9942176938056946 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6782194972038269 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3266999125480652 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.3928571343421936 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11938638240098953 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7301154732704163 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.11558788269758224 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09610409289598465 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *particle/sensitivity‚0.586283802986145 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7678698301315308 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.944444477558136 *jet/accuracy_1_of_4‚0.9464741349220276 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5850287675857544 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.683887779712677 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15993353724479675 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34090909361839294 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5933955311775208 *jet/accuracy_1_of_1‚0.2539796233177185 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0694444477558136 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5161901712417603 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8796935677528381 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9965655207633972 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4831336438655853 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.113216333091259 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9894274473190308 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8333333730697632 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.606765627861023 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10029184818267822 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6957405805587769 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14021500945091248 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.8974359035491943 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7159558534622192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19202573597431183 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5511654019355774 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.33771270513534546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.11273316293954849 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4166666865348816 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.044601697474718094 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9802117347717285 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13879276812076569 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 *jet/accuracy_1_of_3‚0.8791176080703735 (particle/accuracy‚0.8284330368041992 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4925667643547058 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35256412625312805 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9702522158622742 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_4‚0.9919642806053162 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882721304893494 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12515857815742493 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0694444477558136 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚ 2555.46890878 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4825811982154846 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3944590985774994 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9965655207633972 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6063946485519409 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4508667588233948 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9865620732307434 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10029184818267822tØFèDžÝ ‚(:“ ÉŒŠÓãûÖ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚277.723' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.57433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÛåCÞ ‚(:¸  ÉŒŠÓŠúô³ network.recv‚194907) disk./.usagePercent‚70.51355086124316 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.742385228474934 cpu‚25.251905376239726 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1285.in‚0$ memory_percent‚81.05594317118326 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1226.out‚0 network.sent‚ 2.343219e+06 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 47.23828125 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2885.80859375 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0$ disk./.usageGB‚1260.853904724121 disk.drbd1552.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.015625% disk.nvme0n1.out‚8812.55322265625 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 4655.69140625 disk.sda2.out‚ 72.8671875" disk.dm-0.out‚8812.55322265625 disk.drbd2041.in‚0 disk.rbd1.out‚ 11.9033203125 disk.drbd1550.out‚0 disk.sda.out‚ 72.8671875·¾ Eß ‚(:º  ØŒŠÓ–Þð³ network.recv‚196082% disk./.usageGB‚1260.8539848327637 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.742385228474934 disk.drbd2078.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0% disk.nvme0n1.out‚8812.86572265625 disk.drbd1552.out‚0 disk.drbd1931.out‚0" disk.dm-0.out‚8812.86572265625 disk.drbd1550.out‚0 network.sent‚ 2.348743e+06 disk.drbd1550.in‚0 disk.drbd1289.out‚0$ memory_percent‚81.05597496032715# proc.memory.rssMB‚ 2885.80859375 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 73.11328125 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51355534133782 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 11.935546875 disk.sda1.out‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 4655.68359375 cpu‚25.251990753178468 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 73.11328125 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1584.out‚0 disk.sda.in‚0.015625 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 47.23828125AÄÀÛà ‚(:’ ØŒŠÓøŒÑ³ gpu.0.gpu‚42.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚285.34' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.11333333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼<¡Ûžá ‚(:“ 猊Ó“éÔ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.852' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.61733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô…ŒDâ ‚(:¹  猊ÓŽ¬ñ³ disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.sda2.in‚0.015625 disk.dm-0.in‚ 47.23828125# proc.memory.rssMB‚ 2885.80859375 disk.drbd1673.in‚0 network.sent‚ 2.348851e+06 disk.drbd1828.in‚0% disk.nvme0n1.out‚9782.82373046875 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 4655.68359375 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.sda.in‚0.015625" disk.dm-0.out‚9782.82373046875 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0$ memory_percent‚81.05597496032715 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 12.029296875 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 73.46484375* proc.memory.percent‚11.742385228474934 network.recv‚196190 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.8502922058105 disk.drbd2041.out‚0 disk.sda2.out‚ 73.46484375 cpu‚25.25197931641861) disk./.usagePercent‚70.51334883030788 disk.sda1.in‚0 disk.drbd1938.in‚0ti^Nžã ‚(:“ öŒŠÓƶ¸³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚232.201' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.40033333333332 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k—Cä ‚(:¸  öŒŠÓ£ÃÖ³ disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 cpu‚25.2522559263263 disk.drbd1552.in‚0 disk.sda.in‚0.015625 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.sda2.out‚ 73.77734375 disk.drbd1289.out‚0" disk.dm-0.out‚9783.09326171875 disk.drbd1965.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 47.23828125) disk./.usagePercent‚70.51334456355104 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.sda.out‚ 73.77734375 network.sent‚ 2.348917e+06 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.742385228474934 disk.nvme0n1.in‚ 47.23828125 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd2111.in‚0$ memory_percent‚81.05597496032715 disk.drbd1584.out‚0 disk.sda2.in‚0.015625 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 4655.68359375 disk.rbd1.out‚ 12.091796875 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 network.recv‚196256% disk./.usageGB‚1260.8502159118652% disk.nvme0n1.out‚9783.09326171875 disk.drbd1938.in‚0 disk.drbd1828.out‚0äü|õCå ‚(:¸  …ŠÓˆÜå³ disk.nvme0n1.in‚ 47.23828125 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1289.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 47.23828125 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1938.out‚0 network.recv‚200041 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0% disk.nvme0n1.out‚9783.63720703125 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.015625) proc.memory.availableMB‚ 4655.68359375) disk./.usagePercent‚70.51334456355104% disk./.usageGB‚1260.8502159118652 disk.drbd2041.in‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1584.out‚0 cpu‚25.252334891026287 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 74.3046875 disk.sda2.in‚0.015625" disk.dm-0.out‚9783.63720703125 network.sent‚ 2.468536e+06 disk.drbd1965.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0$ memory_percent‚81.05597496032715* proc.memory.percent‚11.742385228474934 disk.sda2.out‚ 74.3046875 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 12.091796875 disk.drbd2118.in‚0ÏÙ‚Åžæ ‚(:“ …ŠÓ©Àų gpu.0.gpu‚55.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.215' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.73833333333333 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÛ“ç ‚(:ˆ ”ŠÓõ™Û³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.879' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.293 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¡F(Eè ‚(:º  ”ŠÓ²Õü³ disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.015625* proc.memory.percent‚11.742385228474934 disk.rbd1.out‚ 12.1240234375 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0% disk./.usageGB‚1260.8502159118652 disk.drbd1356.in‚0 cpu‚25.252318260407467% disk.nvme0n1.out‚9783.91943359375 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.dm-0.in‚ 47.23828125# proc.memory.rssMB‚ 2885.80859375 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1510.in‚0) disk./.usagePercent‚70.51334456355104 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 74.55078125 disk.drbd1289.out‚0 network.sent‚ 2.468644e+06" disk.dm-0.out‚9783.91943359375 disk.sda.in‚0.015625 disk.sda.out‚ 74.55078125 disk.drbd1931.out‚0$ memory_percent‚81.05599085489908 disk.drbd1828.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0( proc.memory.availableMB‚ 4655.6796875 network.recv‚200149 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0°åž2é - _wandbruntime‚2645  _runtime‚2645¹)½Éê à ' lr-AdamW/pg1‚0.00011504342601494784 ' lr-AdamW/pg2‚0.00011504342601494784 trainer/global_step‚1449 % _timestamp‚1.7848418825395412e+09 _runtime‚2645.896798715 _step‚117ŠðxŽÍë Ç ' lr-AdamW/pg1‚0.00011504342601494784 ' lr-AdamW/pg2‚0.00011504342601494784 trainer/global_step‚1449 % _timestamp‚1.7848418825395412e+09 _runtime‚2645.896798715 _step‚117u‹)NA2ì - _wandbruntime‚2645  _runtime‚26455̯ð'í êO RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116238176078336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417309938980864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12583724 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418655215838476 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚418823317491200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 $ _timestamp‚1.784841882540636e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13455801 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4424845 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1512033074176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1851384664576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116237754622976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1509500346086 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417309912807424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1512033074176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3867163 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1851331300352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13465865 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417309938980864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚418821945881600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13465865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26039519 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1513378510336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12583724 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚418823317491200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114386791413760 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8292356 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3867168 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418821945881600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417309912807424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1513378510336 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8292008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚417144370881104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚118 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26049589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12583718 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12583718 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1510844957372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005»ªü î îO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417309912807424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1512033074176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3867163 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1851331300352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13465865 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417309938980864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚418821945881600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13465865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26039519 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1513378510336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12583724 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚418823317491200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114386791413760 YIDeviceStatsM· –øonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8292356 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3867168 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418821945881600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417309912807424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1513378510336 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8292008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚417144370881104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚118 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26049589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12583718 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12583718 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1510844957372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116238176078336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417309938980864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12583724 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418655215838476 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚418823317491200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 $ _timestamp‚1.784841882540636e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13455801 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4424845 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1512033074176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1851384664576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116237754622976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1509500346086 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0v_I/2ï - _wandbruntime‚2647  _runtime‚2647"^,4ú&ð ôM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1511865759124 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚1449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13464883 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8297565 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417588822097920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114465009688576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26067052 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417588822097920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13474961 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8297919 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3869803 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26056968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1514401025536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13464883 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12592085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1852564513792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚418933668374638 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417588848271360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1513055582208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417588848271360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114465377779712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417423147226912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419103249296896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4427762 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1514401025536 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12592085 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26067052 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116318004039168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116317574202368 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418935038414004 $ _timestamp‚1.784841884021942e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419103249296896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419101877680128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12592091 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1852626259456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3869808qòÎçþ&ñ øM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1510521147726 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3869808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1511865759124 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚1449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13464883 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8297565 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417588822097920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114465009688576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26067052 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417588822097920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13474961 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8297919 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3869803 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26056968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1514401025536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13464883 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12592085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1852564513792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚418933668374638 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417588848271360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1513055582208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417588848271360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114465377779712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417423147226912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419103249296896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4427762 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1514401025536 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12592085 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26067052 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116318004039168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116317574202368 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418935038414004 $ _timestamp‚1.784841884021942e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419103249296896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419101877680128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12592091 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1852626259456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8571043 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1564568066560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431255377332224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13002074 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26923891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚432645038370830 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 _step‚122 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432818599962624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1913826122240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1561948840180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚363 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1564568066560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13921817 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13911753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118315741620224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432646408410084 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚358 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120229567742464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120229991294976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118316109711360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3999168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432818599962624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8571406 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431255377332224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4571880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1563222630400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431084459569904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13002074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1563222630400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432819971572224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2732.616492488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26913821 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚431255403505664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13911753 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1913881583616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8571043 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1564568066560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431255377332224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13002074 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26923891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚432645038370830 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 _step‚122 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432818599962624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1913826122240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1561948840180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚363 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1564568066560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13921817 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13911753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118315741620224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432646408410084 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚358 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120229567742464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120229991294976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118316109711360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3999168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432818599962624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8571406 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431255377332224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4571880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1563222630400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431084459569904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13002074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1563222630400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432819971572224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528z±Ö¬+2‡ - _wandbruntime‚2734  _runtime‚2734£¹œ • ˆ õM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120309398720000 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 % _timestamp‚1.7848419707373176e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13010435 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431534286622720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1564245138432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚2734.094936344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118394327986176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431363261343680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13920835 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚432924860946246 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431534312796160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431363235915712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1561625030534 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26931270 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13010435 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4575138 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4574781 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26941354 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431534286622720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1564245138432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1565590581760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚123 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13010441 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚433099903377920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433098531761152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13010441 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8576584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1562969641932 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1565590581760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26941354 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13920835 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1499 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13930913 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118394696077312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ŸÐASÿ&‰ ùM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1499 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13930913 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118394696077312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120309398720000 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 % _timestamp‚1.7848419707373176e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13010435 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431534286622720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1564245138432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚2734.094936344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118394327986176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431363261343680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13920835 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚432924860946246 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431534312796160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431363235915712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1561625030534 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26931270 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13010435 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4575138 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4574781 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26941354 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431534286622720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1564245138432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1565590581760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚123 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13010441 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚433099903377920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433098531761152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13010441 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8576584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1562969641932 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1565590581760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26941354 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13920835 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5{3"àU2Š - _wandbruntime‚2734  _runtime‚2734ú_ 4‹ ‚ /loss/FRt1/detection_loss‚0.5937943396661353 1loss/SRqqt1/detection_loss‚1.3054217076324939 /loss/FBt2/detection_loss‚0.6927946836129442 epoch‚5 % _timestamp‚1.7848419707743661e+09 2loss/SRqqt2/assignment_loss‚0.7071178632527739 1loss/FBt2/assignment_loss‚0.12176624792937804 &loss/total_loss‚15.014885106344536 trainer/global_step‚1499 0loss/FRt2/assignment_loss‚1.7570284622557022 1loss/FBt1/assignment_loss‚0.21129637694155629 /loss/FBt1/detection_loss‚3.4281753332100133 _runtime‚2734.131558661 _step‚124 2loss/SRqqt1/assignment_loss‚0.9812000916478993 /loss/FRt2/detection_loss‚1.4708634870417994 1loss/SRqqt2/detection_loss‚0.7818783633968174 0loss/FRt1/assignment_loss‚2.9635481497570244IÑÜ Œ † 1loss/SRqqt2/detection_loss‚0.7818783633968174 0loss/FRt1/assignment_loss‚2.9635481497570244 /loss/FRt1/detection_loss‚0.5937943396661353 1loss/SRqqt1/detection_loss‚1.3054217076324939 /loss/FBt2/detection_loss‚0.6927946836129442 epoch‚5 % _timestamp‚1.7848419707743661e+09 2loss/SRqqt2/assignment_loss‚0.7071178632527739 1loss/FBt2/assignment_loss‚0.12176624792937804 &loss/total_loss‚15.014885106344536 trainer/global_step‚1499 0loss/FRt2/assignment_loss‚1.7570284622557022 1loss/FBt1/assignment_loss‚0.21129637694155629 /loss/FBt1/detection_loss‚3.4281753332100133 _runtime‚2734.131558661 _step‚124 2loss/SRqqt1/assignment_loss‚0.9812000916478993 /loss/FRt2/detection_loss‚1.4708634870417994|<Ÿ ‡: ‚(:¯  ýŠÓò£ê³ network.recv‚207578 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0) proc.memory.availableMB‚ 4655.73046875 disk.nvme0n1.in‚ 47.23828125 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.015625 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.sda.out‚ 77.3359375$ memory_percent‚81.05578422546387* proc.memory.percent‚11.742385228474934% disk./.usageGB‚1260.8500518798828 disk.sda2.out‚ 77.3359375 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0# disk.nvme0n1.out‚9787.576171875 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 47.23828125 disk.dm-0.out‚9787.576171875 disk.drbd1965.out‚0 network.sent‚ 2.605002e+06 disk.drbd2078.out‚0 disk.rbd1.out‚12.46875 disk.sda.in‚0.015625 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0( disk./.usagePercent‚70.5133353900239 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 cpu‚25.253043124259854ˆ”ŸŽ ‚(:” ýŠÓյų gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚112.588' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.529333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01}ž ‚(:“ ŒŽŠÓŽ×ß³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚291.221' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.07366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô¨QA ‚(:¶  ŒŽŠÓÂ×ü³ disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51334819029435 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚9787.783203125 cpu‚25.253024077943316 network.recv‚207644 disk.sda.out‚ 77.57421875 disk.dm-0.in‚ 47.23828125* proc.memory.percent‚11.742385228474934 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 47.23828125( proc.memory.availableMB‚ 5390.6640625 disk.drbd1552.in‚0 disk.sda2.out‚ 77.57421875$ memory_percent‚78.06533177693686 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.015625 disk.dm-0.out‚9787.783203125 disk.drbd1226.out‚0 disk.sda.in‚0.015625 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.rbd1.out‚ 12.5009765625 disk.drbd1550.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0% disk./.usageGB‚1260.8502807617188 disk.sda1.in‚0 disk.drbd1289.out‚0 network.sent‚ 2.605068e+06 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0¥`]D‘ ‚(:¹  ›ŽŠÓЛ…´ network.sent‚ 2.605134e+06% disk./.usageGB‚1260.8502807617188 disk.drbd2078.out‚0$ memory_percent‚79.27155494689941 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 47.23828125% disk.nvme0n1.out‚9788.21533203125 disk.sda1.in‚0 cpu‚25.2533626888422) disk./.usagePercent‚70.51334819029435 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 network.recv‚207710 disk.drbd1289.in‚0" disk.dm-0.out‚9788.21533203125 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 5094.22265625 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 77.92578125 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.015625 disk.drbd1965.out‚0* proc.memory.percent‚11.742385228474934 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 12.7744140625 disk.drbd1817.out‚0 disk.sda.out‚ 77.92578125 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 47.23828125 disk.drbd1965.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0Òÿè‹Ÿ’ ‚(:” ›ŽŠÓ½©é³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚301.754' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.58466666666668 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø¡‚ž“ ‚(:“ ªŽŠÓÄï²³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.388' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.46266666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÆîIE” ‚(:º  ªŽŠÓ£Áг disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.drbd2111.out‚0% disk.nvme0n1.out‚9788.48876953125 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 network.sent‚ 2.675672e+06 disk.drbd2078.out‚0 disk.drbd1552.out‚0 cpu‚25.253352537261875 proc.cpu.threads‚17$ disk./.usageGB‚1260.850299835205 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 78.32421875 disk.drbd1931.in‚0) disk./.usagePercent‚70.51334925698356 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1510.out‚0 disk.drbd1550.in‚0$ memory_percent‚80.44711748758951 disk.drbd2078.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 47.23828125 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0) proc.memory.availableMB‚ 4805.31640625# proc.memory.rssMB‚ 2885.80859375 disk.drbd1828.in‚0 disk.drbd1584.out‚0" disk.dm-0.out‚9788.48876953125 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 78.32421875 disk.drbd1289.in‚0 disk.drbd2041.out‚0 network.recv‚210686 disk.sda.in‚0.015625 disk.drbd1938.out‚0* proc.memory.percent‚11.742385228474934 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 12.8369140625 disk.drbd1673.in‚0ÔÏXÍ• ‚(:’ ¹ŽŠÓæÞγ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.87' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.95666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1615757622784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1613052722988 $ _timestamp‚1.784842056382425e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14377769 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445200841857024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4131168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446816625653248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446816625653248 RCDeviceStatsMonitor.on_train_batch_start/allocated_Üî¡õ€bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1615757622784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4131163 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124221354467840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚2819.739999516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13420418 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445200868030464 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8850109 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8849747 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122245059917824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445024522830736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1614412186624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27798193 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1976347914240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445200868030464 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27788123 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13420424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14367705 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13420418 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚446636230942438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1614412186624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124221775923200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 trainer/global_step‚1549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446815254043648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚446815254043648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13420424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655ÁÀºÙô'œ îO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446815254043648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚446815254043648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13420424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4718584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4718941 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122245428008960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚362 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1611708111702 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1615757622784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1613052722988 $ _timestamp‚1.784842056382425e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14377769 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445200841857024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4131168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446816625653248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446816625653248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1615757622784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4131163 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124221354467840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚2819.739999516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13420418 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445200868030464 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8850109 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8849747 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122245059917824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445024522830736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1614412186624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27798193 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1976347914240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445200868030464 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27788123 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13420424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14367705 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13420418 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚446636230942438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1614412186624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124221775923200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 trainer/global_step‚1549~ŽqUþœ ‚(:‘ ÈŽŠÓÊó׳ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.917' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.97233333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»íØm?ž ‚(:´  ÈŽŠÓ‹÷÷³% disk./.usageGB‚1260.8502807617188 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 79.0859375 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2885.80859375* proc.memory.percent‚11.742385228474934 disk.drbd1550.in‚0 disk.sda1.in‚0" disk.dm-0.out‚9788.94970703125 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 79.0859375 disk.drbd1289.in‚0 cpu‚25.25374966098183) disk./.usagePercent‚70.51334819029435 disk.drbd1673.in‚0 disk.drbd1584.out‚0 network.sent‚ 2.748922e+06 network.recv‚213427 disk.dm-0.in‚ 47.23828125 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0% disk.nvme0n1.out‚9788.94970703125 disk.drbd1817.out‚0& proc.memory.availableMB‚ 4374.53125 disk.sda2.in‚0.015625 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.rbd1.out‚ 12.869140625 disk.drbd1550.out‚0 disk.drbd2118.out‚0$ memory_percent‚82.19998677571616 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 47.23828125 disk.sda.in‚0.015625 disk.rbd1.in‚0 proc.cpu.threads‚17à®2Ÿ - _wandbruntime‚2821  _runtime‚2821}Çx°ú&  ôM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14376787 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445303350032480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447095185842176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445303324604512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14376787 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14386865 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1616780137984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447096557458944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445479751147520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124301618807296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445479751147520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4133803 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚127 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13428785 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124301197359104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1549 $ _timestamp‚1.784842057855293e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8855228 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4133808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚446916053517854 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1615434694656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27815656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14386865 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122324014374912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1977551075328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚446917423557220 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27805572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1615434694656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13428785 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122323646283776 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1616780137984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1977604432384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4721770 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1612728913342 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445479777320960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚2821.212883694 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8855578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27805572¶fó•T¡ øM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13428785 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124301197359104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1549 $ _timestamp‚1.784842057855293e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8855228 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4133808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚446916053517854 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1615434694656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27815656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14386865 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122324014374912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1977551075328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚446917423557220 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27805572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1615434694656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13428785 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122323646283776 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1616780137984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1977604432384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4721770 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1612728913342 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445479777320960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚2821.212883694 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8855578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27805572 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14376787 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445303350032480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447095185842176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445303324604512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14376787 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14386865 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1616780137984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447096557458944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445479751147520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124301618807296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445479751147520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4133803 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚127 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¤8î2¢ - _wandbruntime‚2821  _runtime‚2821B¯vK£ ý 1loss/SRqqt1/detection_loss‚1.2698527366125123 /loss/FBt1/detection_loss‚2.2102871641516684 epoch‚5 2loss/SRqqt1/assignment_loss‚1.0161902974083972 .loss/FBt2/detection_loss‚4.121629858229455 /loss/FRt1/assignment_loss‚2.897482456633193 /loss/FRt2/detection_loss‚1.4214481514583834 &loss/total_loss‚17.099835262391334 trainer/global_step‚1549 _runtime‚ 2821.25336937 _step‚128 2loss/SRqqt2/assignment_loss‚0.7577081386011119 0loss/FBt1/assignment_loss‚0.1694581969170031 1loss/FBt2/assignment_loss‚0.12221041552856564 /loss/FRt1/detection_loss‚0.6452519991459714 1loss/SRqqt2/detection_loss‚0.8040115469089185 % _timestamp‚1.7848420578961651e+09 /loss/FRt2/assignment_loss‚1.664304300796154~¦(¯¤ ‚ 1loss/SRqqt2/detection_loss‚0.8040115469089185 % _timestamp‚1.7848420578961651e+09 /loss/FRt2/assignment_loss‚1.664304300796154 1loss/SRqqt1/detection_loss‚1.2698527366125123 /loss/FBt1/detection_loss‚2.2102871641516684 epoch‚5 2loss/SRqqt1/assignment_loss‚1.0161902974083972 .loss/FBt2/detection_loss‚4.121629858229455 /loss/FRt1/assignment_loss‚2.897482456633193 /loss/FRt2/detection_loss‚1.4214481514583834 &loss/total_loss‚17.099835262391334 trainer/global_step‚1549 _runtime‚ 2821.25336937 _step‚128 2loss/SRqqt2/assignment_loss‚0.7577081386011119 0loss/FBt1/assignment_loss‚0.1694581969170031 1loss/FBt2/assignment_loss‚0.12221041552856564 /loss/FRt1/detection_loss‚0.6452519991459714€Öe¹X“¥ ‚(:ˆ ׎ŠÓÛÛ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚286.425' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×'@¦ ‚(:µ  ׎ŠÓÅŒø³ network.recv‚213493 disk.drbd1285.in‚0 disk.sda2.out‚ 79.4140625 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.742385228474934) disk./.usagePercent‚70.51334819029435 disk.drbd1552.out‚0( proc.memory.availableMB‚ 4102.0546875$ disk.nvme0n1.out‚9789.3232421875 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0$ memory_percent‚83.30869674682617 disk.drbd1931.in‚0 disk.drbd1938.in‚0! disk.dm-0.out‚9789.3232421875 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8502807617188 disk.drbd1550.in‚0 disk.sda.in‚0.015625 cpu‚25.253730020864328 disk.drbd1828.out‚0 disk.sda2.in‚0.015625 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 47.23828125 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 12.962890625 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.748988e+06 disk.sda1.in‚0 disk.sda.out‚ 79.41406250âó-ž§ ‚(:“ 掊Óðÿá³ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚260.694' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.89800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0feSR<¨ ‚(:±  掊ÓÁÙ´ disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0$ memory_percent‚84.25583839416504* proc.memory.percent‚11.742385228474934 network.recv‚214407% disk./.usageGB‚1260.8502807617188 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 47.23828125 disk.drbd1828.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1550.out‚0 disk.sda2.in‚0.015625 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0) proc.memory.availableMB‚ 3869.28515625 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0! disk.dm-0.out‚9789.8740234375# proc.memory.rssMB‚ 2885.80859375 cpu‚25.253795978196482 disk.sda.in‚0.015625 disk.drbd1673.out‚0 disk.sda2.out‚79.71875 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚9789.8740234375 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17 network.sent‚ 2.75205e+06 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.sda.out‚79.71875) disk./.usagePercent‚70.51334819029435 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 13.025390625 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0Ì¿NGœ© ‚(:‘ õŽŠÓ‚Ïã³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.904' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.63466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙDDDª ‚(:¹  õŽŠÓÇÉ‚´ disk.sda1.out‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51297100899173 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0" disk.dm-0.out‚9791.08544921875 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.sda.in‚0.015625 disk.drbd1673.in‚0 disk.sda2.out‚ 80.14453125 disk.drbd1931.in‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚9791.08544921875 disk.drbd1552.in‚0% disk./.usageGB‚1260.8435363769531 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.015625 disk.drbd1289.out‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 3683.3359375 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 80.14453125 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.742385228474934 disk.dm-0.in‚ 47.23828125 disk.drbd2078.in‚0 disk.drbd1550.in‚0$ memory_percent‚85.01246770222983 network.sent‚ 2.752158e+06 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 network.recv‚214515 disk.drbd2041.in‚0 disk.drbd1285.out‚0 cpu‚25.254034987121116 disk.rbd1.out‚ 13.025390625 disk.drbd1552.out‚0rK¹C« ‚(:¸  „ŠÓ±€´ disk.drbd1931.in‚0" disk.dm-0.out‚9791.27294921875 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1817.out‚0 disk.sda.in‚0.015625 disk.dm-0.in‚ 47.23828125 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0$ memory_percent‚85.01246770222983 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚9791.27294921875 disk.drbd1510.out‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51297100899173 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 network.recv‚214557 disk.drbd1356.in‚0 disk.sda1.out‚0* proc.memory.percent‚11.742385228474934 proc.cpu.threads‚17 disk.sda.out‚ 80.37890625 disk.sda2.out‚ 80.37890625 network.sent‚ 2.7522e+06 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 3683.3359375 cpu‚25.254186077718916 disk.sda2.in‚0.015625 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0% disk./.usageGB‚1260.8435363769531 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 13.0576171875 disk.sda1.in‚0 _äñŸ¬ ‚(:” „ŠÓüŒÜ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚306.342' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.11399999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çèa“­ ‚(:ˆ “ŠÓñ‰³³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚245.022' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚81.674 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v‰âF<® ‚(:±  “ŠÓûŸÐ³ disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.sda2.in‚0.015625 disk.drbd1356.out‚0 cpu‚25.254598228929336 disk.rbd1.out‚ 13.1513671875 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8437118530273 disk.sda.in‚0.015625 disk.drbd1938.in‚0) proc.memory.availableMB‚ 3683.33203125 disk.drbd1285.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda.out‚80.71875 disk.drbd1931.out‚0 disk.drbd1965.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1931.in‚0 disk.drbd2041.in‚0$ memory_percent‚85.01248359680176 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚80.71875 disk.drbd2078.in‚0* proc.memory.percent‚11.742385228474934) disk./.usagePercent‚70.51298082253241 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 network.sent‚ 2.8176e+06 disk.drbd1817.in‚0 disk.dm-0.in‚ 47.23828125! disk.dm-0.out‚9791.8896484375 disk.drbd2078.out‚0 network.recv‚216881 disk.drbd1552.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚ 47.23828125 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0$ disk.nvme0n1.out‚9791.8896484375 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0’šÕ2¯ - _wandbruntime‚2906  _runtime‚2906ù$Y³É° à % _timestamp‚1.7848421431712403e+09 _runtime‚2906.528474177 _step‚129 ' lr-AdamW/pg1‚0.00011447819328690167 ' lr-AdamW/pg2‚0.00011447819328690167 trainer/global_step‚1599P⨠α È ' lr-AdamW/pg1‚0.00011447819328690167 ' lr-AdamW/pg2‚0.00011447819328690167 trainer/global_step‚1599 % _timestamp‚1.7848421431712403e+09 _runtime‚2906.528474177 _step‚129áºi32² - _wandbruntime‚2906  _runtime‚2906Vw•ñ'³ ëO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459146306381824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1662811994510 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128213728739840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4263163 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13838774 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9128548 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚1599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460811908124672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28672495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128213300993024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13838768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4263168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460627423514046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460813279734272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14823657 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4865748 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126174378215424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460628793553300 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13838768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1666947179008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14833721 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2038922777600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _runtime‚2906.532928118 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1665601742848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2038982433280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458964611519536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1666947179008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460811908124672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126174746306560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1665601742848 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9128916 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4865385 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚458964636947504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460813279734272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459146306381824 % _timestamp‚1.7848421431722713e+09ÜA^c´ ðO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13838774 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459146306381824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1662811994510 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128213728739840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4263163 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13838774 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9128548 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚1599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460811908124672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28672495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128213300993024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 11YÉ ¶“23180544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13838768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4263168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460627423514046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460813279734272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14823657 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4865748 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126174378215424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460628793553300 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13838768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1666947179008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14833721 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2038922777600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _runtime‚2906.532928118 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1665601742848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2038982433280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458964611519536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1666947179008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460811908124672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126174746306560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1665601742848 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9128916 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4865385 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚458964636947504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460813279734272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459146306381824 % _timestamp‚1.7848421431722713e+09 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9134119 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1667969694208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28679874 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460908616128828 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848421446510448e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14832739 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚461093211539968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14842817 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1667969694208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459243438721280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126252964581376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1666624250880 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126253332672512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚461093211539968 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13847135 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28689958 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459425215672320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460907246089462 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14842817 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459425241845760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14832739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2040183459840 _runtime‚2908.008659261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28679874 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459425215672320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚131°('· úM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚461093211539968 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13847135 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28689958 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459425215672320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460907246089462 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14842817 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459425241845760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14832739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2040183459840 _runtime‚2908.008659261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28679874 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459425215672320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚131 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28689958 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9134119 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1667969694208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28679874 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460908616128828 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848421446510448e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14832739 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚461093211539968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14842817 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1667969694208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459243438721280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126252964581376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1666624250880 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126253332672512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ƒøé©2¸ - _wandbruntime‚2908  _runtime‚2908€D‹Ù¹ þ 1loss/FBt2/assignment_loss‚0.08434276476576637 /loss/FRt1/detection_loss‚0.7045959151970003 1loss/SRqqt2/detection_loss‚0.6883845133758871 _runtime‚2908.042393891 &loss/total_loss‚14.526890787722408 % _timestamp‚1.7848421446851623e+09 1loss/SRqqt1/assignment_loss‚0.889084317142676 0loss/FBt1/assignment_loss‚0.2011454000060136 1loss/SRqqt1/detection_loss‚1.3533591444138438 .loss/FBt1/detection_loss‚2.996096695959568 epoch‚5 /loss/FRt2/detection_loss‚1.3877800288487803 .loss/FBt2/detection_loss‚0.978631688274425 trainer/global_step‚1599 _step‚132 0loss/FRt1/assignment_loss‚2.9350674782000796 0loss/FRt2/assignment_loss‚1.6161280420806765 2loss/SRqqt2/assignment_loss‚0.6922747994576925#Öÿª º ƒ 0loss/FRt1/assignment_loss‚2.9350674782000796 0loss/FRt2/assignment_loss‚1.6161280420806765 2loss/SRqqt2/assignment_loss‚0.6922747994576925 1loss/FBt2/assignment_loss‚0.08434276476576637 /loss/FRt1/detection_loss‚0.7045959151970003 1loss/SRqqt2/detection_loss‚0.6883845133758871 _runtime‚2908.042393891 &loss/total_loss‚14.526890787722408 % _timestamp‚1.7848421446851623e+09 1loss/SRqqt1/assignment_loss‚0.889084317142676 0loss/FBt1/assignment_loss‚0.2011454000060136 1loss/SRqqt1/detection_loss‚1.3533591444138438 .loss/FBt1/detection_loss‚2.996096695959568 epoch‚5 /loss/FRt2/detection_loss‚1.3877800288487803 .loss/FBt2/detection_loss‚0.978631688274425 trainer/global_step‚1599 _step‚132„Tc#ž» ‚(:“ ¢ŠÓÛŽÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.428' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.14266666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q¡yR?¼ ‚(:´  ¢ŠÓ‘Ãù³ disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.742385228474934% disk./.usageGB‚1260.8438148498535 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.sda.in‚0.015625 disk.drbd1931.in‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚9792.0693359375 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 network.sent‚ 2.825866e+06 disk.drbd1817.in‚0 disk.drbd1938.out‚0 cpu‚25.254568980928937 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.sda2.in‚0.015625 disk.drbd1584.in‚0 disk.sda2.out‚ 81.0546875! disk.dm-0.out‚9792.0693359375 disk.drbd1938.in‚0 disk.rbd1.out‚ 13.2138671875 disk.drbd1817.out‚0 disk.drbd1289.out‚0& proc.memory.availableMB‚ 3683.34375 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51298658265411 disk.drbd1550.in‚0 disk.sda.out‚ 81.0546875$ memory_percent‚85.01243591308594# proc.memory.rssMB‚ 2885.80859375 disk.nvme0n1.in‚ 47.23828125 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 47.23828125 disk.drbd2041.in‚0 network.recv‚217912 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0“M"”½ ‚(:‰ ±ŠÓž™º³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.568' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.856 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë7ÜsE¾ ‚(:º  ±ŠÓÕ©Ú³ network.sent‚ 2.825932e+06 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.02734375" disk.dm-0.out‚9792.41455078125 disk.drbd1285.in‚0% disk.nvme0n1.out‚9792.41455078125 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0& proc.memory.availableMB‚ 3683.34375 network.recv‚217978 disk.sda2.in‚ 0.02734375 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 81.6015625 disk.drbd1289.in‚0 cpu‚25.254634738377653 disk.rbd1.out‚ 13.2138671875 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51298658265411 disk.drbd2078.out‚0 disk.drbd1510.out‚0$ memory_percent‚85.01243591308594 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.sda2.out‚ 81.6015625 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8438148498535 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 47.61328125* proc.memory.percent‚11.742385228474934 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 47.61328125ìðd`¿ ‚(:’ ÀŠÓ´¥Ì³ gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.457' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.48566666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fêºSÀ ‚(:»  ÀŠÓŠ´ê³ disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 81.83984375 disk.drbd2078.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 47.61328125 cpu‚25.2549453897299% disk.nvme0n1ºþ_ó.out‚9792.64697265625 disk.drbd1552.out‚0* proc.memory.percent‚11.742385228474934 disk.drbd1285.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 47.61328125 disk.sda.in‚ 0.02734375 disk.drbd1289.in‚0 disk.drbd1965.out‚0 network.sent‚ 2.828906e+06 disk.sda2.out‚ 81.83984375 disk.drbd1938.in‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51298764934332 disk.drbd1510.out‚0 disk.drbd1552.in‚0" disk.dm-0.out‚9792.64697265625 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 network.recv‚218947 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.sda2.in‚ 0.02734375$ memory_percent‚85.01521746317545% disk./.usageGB‚1260.8438339233398 disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0) proc.memory.availableMB‚ 3682.66015625# proc.memory.rssMB‚ 2885.80859375 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1817.in‚0 disk.rbd1.out‚ 13.24609375l¨˜FžÁ ‚(:“ ÏŠÓ£Ûdz gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.475' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.15833333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gœA ‚(:¶  ÏŠÓ¥üé³ disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.sda2.out‚ 82.5546875! disk.dm-0.out‚9793.2138671875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 3682.5859375* proc.memory.percent‚11.742385228474934 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 13.33984375 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 network.recv‚219013% disk./.usageGB‚1260.8439559936523 disk.sda1.in‚0 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1552.out‚0 disk.nvme0n1.in‚ 48.0078125 disk.drbd1550.in‚0) disk./.usagePercent‚70.51299447615422 disk.sda2.in‚ 0.02734375 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.dm-0.in‚ 48.0078125 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 cpu‚25.254836995701808 network.sent‚ 2.828972e+06 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 82.5546875 disk.drbd1965.out‚0$ memory_percent‚85.01551946004233 disk.drbd1289.out‚0 disk.drbd1965.in‚0$ disk.nvme0n1.out‚9793.2138671875 disk.drbd1510.out‚0 disk.sda.in‚ 0.02734375 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0r]/€žà ‚(:“ ÞŠÓ°­¾³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.925' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.64166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FšcF<Ä ‚(:±  ÞŠÓàÑÞ³ disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda.out‚ 82.85546875 disk.drbd1550.out‚0 network.sent‚ 2.82908e+06% disk./.usageGB‚1260.8439750671387 disk.sda.in‚ 0.02734375 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.dm-0.out‚ 9793.5546875 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51299554284343 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd1938.in‚0 disk.drbd2078.in‚0! disk.nvme0n1.out‚ 9793.5546875 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0# proc.memory.rssMB‚ 2885.80859375* proc.memory.percent‚11.742385228474934 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 13.40234375 disk.nvme0n1.in‚ 51.5234375 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 51.5234375 disk.drbd2078.out‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 3682.50390625 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.sda2.out‚ 82.85546875 disk.drbd1289.in‚0 cpu‚25.25498085991006 network.recv‚219121 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0$ memory_percent‚85.01585324605307 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0\sß|‘Å ‚(:† íŠÓ¸Äê³ gpu.0.gpu‚6.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚140.208' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚46.736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÉ;Æ ‚(:°  íŠÓÞý‹´ disk.drbd1828.in‚0# proc.memory.rssMB‚ 2885.80859375 disk.drbd1584.in‚0% disk.nvme0n1.out‚9794.05126953125 network.recv‚219987 disk.drbd1552.in‚0 disk.rbd1.out‚ 13.40234375 network.sent‚ 2.832162e+06 disk.drbd1673.in‚0 disk.sda2.out‚ 83.3984375 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.742385228474934 disk.drbd1226.in‚0" disk.dm-0.out‚9794.05126953125 disk.drbd2118.out‚0 disk.dm-0.in‚ 51.5234375 disk.drbd1965.out‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8439559936523 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 51.5234375 disk.drbd1550.out‚0 disk.sda.out‚ 83.3984375 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 cpu‚25.255529610353733 disk.sda2.in‚ 0.02734375 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0" memory_percent‚85.015869140625 disk.sda.in‚ 0.02734375 disk.drbd1356.in‚0" proc.memory.availableMB‚3682.5 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.51299447615422 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0½/¡2Ç - _wandbruntime‚2993  _runtime‚299307BÉÈ Ã _step‚133 ' lr-AdamW/pg1‚0.00011427509047834735 ' lr-AdamW/pg2‚0.00011427509047834735 trainer/global_step‚1649 % _timestamp‚1.7848422301200254e+09 _runtime‚2993.477254058iàUXÎÉ È ' lr-AdamW/pg2‚0.00011427509047834735 trainer/global_step‚1649 % _timestamp‚1.7848422301200254e+09 _runtime‚2993.477254058 _step‚133 ' lr-AdamW/pg1‚0.00011427509047834735…?ÛYß2Ê - _wandbruntime‚2993  _runtime‚2993ÞÒ]Èñ'Ë ëO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5011344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474618616085654 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1713915877318 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132205579190784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474809933815296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473091770906624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15289673 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15279609 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚1649 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474808562205696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29536727 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474808562205696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1718136735232 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9406874 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2101514586624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848422301210518e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473091797080064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚474619986124908 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1718136735232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1715260488604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1716791299072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2993.478581742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4395168 _step‚134 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130104064604160 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472904700208336"†\Žö'Ì ðO L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9406507 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132205164026880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15289673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14257118 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474808562205696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29536727 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474808562205696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1718136735232 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9406874 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2101514586624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848422301210518e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473091797080064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚474619986124908 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1718136735232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1715260488604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1716791299072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2993.478581742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4395168 _step‚134 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130104064604160 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472904700208336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5011344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474618616085654 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1713915877318 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132205579190784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474809933815296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473091770906624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15289673 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15279609 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚1649 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473370706370560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473370706370560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9411981 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14265485 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1717813807104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15288691 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473370680197120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5014178 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14265491 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132285416611328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130182282878976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474899808700436 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2955417?è_ð6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1717813807104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29564260 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4397808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132284995163136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15298769 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1719159250432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15288691 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475089865620992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14265485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1714936678958 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚473370680197120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14265491 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130182650970112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚135 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29564260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2994.953404256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475088494004224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚473183501982112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1714936678958 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚473370680197120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14265491 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130182650970112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚135 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29564260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2994.953404256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475088494004224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚473183501982112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473370706370560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473370706370560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9411981 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14265485 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1717813807104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15288691 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473370680197120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5014178 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14265491 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132285416611328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130182282878976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474899808700436 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29554176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1717813807104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29564260 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4397808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132284995163136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15298769 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1719159250432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15288691 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475089865620992‡ à\2Ð - _wandbruntime‚2994  _runtime‚2994ö)o@Ñ € /loss/FBt1/detection_loss‚1.6706627255369877 epoch‚5 trainer/global_step‚1649 % _timestamp‚1.7848422316351106e+09 _runtime‚2994.992311282 0loss/FRt1/assignment_loss‚2.8786104067166645 0loss/FBt2/assignment_loss‚0.0894422355424586 /loss/FRt2/assignment_loss‚1.481721207165896 2loss/SRqqt1/assignment_loss‚0.7465756074449575 _step‚136 1loss/FBt1/assignment_loss‚0.45563365612881546 1loss/SRqqt1/detection_loss‚1.2508182094860154 1loss/SRqqt2/detection_loss‚0.7705501869470017 /loss/FBt2/detection_loss‚0.9198752758878872 &loss/total_loss‚13.060811515752789 2loss/SRqqt2/assignment_loss‚0.6494946129217408 /loss/FRt1/detection_loss‚0.6685079492134456 /loss/FRt2/detection_loss‚1.4789194427609187oƒ²l Ò … 0loss/FBt2/assignment_loss‚0.0894422355424586 /loss/FRt2/assignment_loss‚1.481721207165896 2loss/SRqqt1/assignment_loss‚0.7465756074449575 _step‚136 1loss/FBt1/assignment_loss‚0.45563365612881546 1loss/SRqqt1/detection_loss‚1.2508182094860154 1loss/SRqqt2/detection_loss‚0.7705501869470017 /loss/FBt2/detection_loss‚0.9198752758878872 &loss/total_loss‚13.060811515752789 2loss/SRqqt2/assignment_loss‚0.6494946129217408 /loss/FRt1/detection_loss‚0.6685079492134456 /loss/FRt2/detection_loss‚1.4789194427609187 /loss/FBt1/detection_loss‚1.6706627255369877 epoch‚5 trainer/global_step‚1649 % _timestamp‚1.7848422316351106e+09 _runtime‚2994.992311282 0loss/FRt1/assignment_loss‚2.8786104067166645ˆ}}WžÓ ‚(:“ üŠÓÈϳ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.886' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.62866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¤ìŠGÔ ‚(:¼  üŠÓћѳ disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1285.out‚0$ memory_percent‚85.01601219177246 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda2.in‚ 0.02734375$ disk./.usageGB‚1260.843936920166 disk.dm-0.in‚ 51.5234375 disk.drbd1828.out‚0" disk.dm-0.out‚9794.28173828125 disk.nvme0n1.in‚ 51.5234375% disk.nvme0n1.out‚9794.28173828125 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 83.73046875 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 network.sent‚ 2.832228e+06 disk.rbd1.out‚ 13.4345703125* proc.memory.percent‚11.742385228474934) disk./.usagePercent‚70.51299340946503 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.02734375 network.recv‚220053 disk.drbd1550.out‚0 cpu‚25.255427956779364 disk.drbd1552.out‚0) proc.memory.availableMB‚ 3682.46484375# proc.memory.rssMB‚ 2885.80859375 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 83.73046875 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17¦lñ‰žÕ ‚(:“ ‹ŠÓá£×³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.585' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.52833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú&+e@Ö ‚(:µ  ‹ŠÓ܉õ³ disk.sda.out‚ 84.03515625 disk.sda2.out‚ 84.03515625 disk.drbd1931.in‚0 disk.drbd1285.out‚0 network.sent‚ 2.832336e+06 disk.sda1.in‚0 disk.drbd1550.in‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0$ disk./.usageGB‚1260.843936920166 disk.drbd1356.in‚0 disk.drbd2041.out‚0 network.recv‚220161 disk.drbd1510.in‚0 disk.drbd1552.in‚0( proc.memory.availableMB‚ 3682.4609375 disk.drbd1817.out‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2885.80859375* proc.memory.percent‚11.742385228474934 disk.sda.in‚ 0.02734375 disk.sda2.in‚ 0.02734375 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.dm-0.out‚ 9794.72265625 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.dm-0.in‚ 51.5234375 disk.drbd1356.out‚0$ memory_percent‚85.01602808634439 cpu‚25.255560392784762 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 51.5234375 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 13.5283203125 disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0" disk.nvme0n1.out‚ 9794.72265625) disk./.usagePercent‚70.51299340946503 disk.drbd1584.out‚0ÿùd Ÿ× ‚(:” šŠÓîɳ gpu.0.gpu‚38.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚109.468' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.489333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Њ?Ø ‚(:´  šŠÓݶè³ disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 network.recv‚222868 disk.sda1.in‚0 disk.dm-0.in‚ 51.5234375 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0$ memory_percent‚85.01604398091635 disk.sda1.out‚0 disk.sda2.out‚ 84.33984375" disk.nvme0n1.out‚ 9795.26171875 disk.drbd1550.out‚0 disk.sda.in‚ 0.02734375 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0# proc.memory.rssMB‚ 2885.80859375* proc.memory.percent‚11.742385228474934 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 51.5234375 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.02734375$ disk./.usageGB‚1260.844181060791 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.dm-0.out‚ 9795.26171875 disk.drbd2078.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 84.33984375 network.sent‚ 2.895095e+06 disk.drbd1584.out‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51300706308683 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 13.5908203125) proc.memory.availableMB‚ 3682.45703125 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1510.out‚0 cpu‚25.2555348076839š|€’Ù ‚(:‡ ©ŠÓ¾“¾³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚316.26' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚105.42 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—×1½AÚ ‚(:¶  ©ŠÓŽ¾Þ³# proc.memory.rssMB‚ 2885.80859375 disk.drbd2118.in‚0 disk.sda2.out‚84.78125 disk.drbd1828.out‚0 disk.drbd1550.in‚0$ memory_percent‚85.01604398091635 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd2041.out‚0( disk./.usagePercent‚70.5130064230733 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 3682.45703125 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd1226.out‚0" disk.dm-0.out‚9795.53369140625 disk.drbd2111.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.742385228474934 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 13.5908203125 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 51.5234375 network.sent‚ 2.895245e+06 network.recv‚223018 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.sda.out‚84.78125 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.02734375 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd2078.out‚0% disk./.usageGB‚1260.8441696166992 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0% disk.nvme0n1.out‚9795.53369140625 cpu‚25.255671556408377 disk.drbd1584.out‚0 disk.dm-0.in‚ 51.5234375 disk.drbd1510.in‚0ñ½á¾Û ‚(:’ ¸ŠÓ̳³ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚314.325' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.77499999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xP}BÜ ‚(:·  ¸ŠÓÿÿÓ³ disk.drbd1510.out‚0% proc.memory.availableMB‚ 3683.5625 disk.drbd1550.in‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8441734313965 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0$ memory_percent‚85.01154581705728# proc.memory.rssMB‚ 2885.80859375 disk.drbd2118.out‚0 disk.sda.out‚ 85.04296875 disk.rbd1.out‚ 13.623046875 network.recv‚223084 disk.drbd1226.out‚0 cpu‚25.26574689729073" disk.dm-0.out‚9795.86962890625 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚9795.86962890625 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.742385228474934 disk.drbd1938.out‚0 disk.dm-0.in‚ 51.5234375 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.sda2.out‚ 85.04296875 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51300663641115 disk.sda.in‚ 0.02734375 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 51.5234375 disk.drbd1285.in‚0 disk.drbd1226.in‚0 network.sent‚ 2.895311e+06 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚06…”O2Ý - _wandbruntime‚3066  _runtime‚3066%w/ŒÑÞ —x >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 *jet/accuracy_1_of_4‚0.9399362802505493 particle/accuracy_1_of_4‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026101233437657356 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6883958578109741 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9583333730697632 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11158985644578934 * Purity/2FRt0SRqqt2FBt/FRt_purity‚0.625 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.22094236314296722 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47725802659988403 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09154290705919266 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08353257179260254 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3706302046775818 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9924100041389465 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7340009808540344 +jet/accuracy_1_of_1‚0.24097482860088348 /particle/accuracy_4_of_4‚0.8886705636978149 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6818182468414307 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6724880933761597 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5403302311897278 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9933681488037109 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9883203506469727 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07533912360668182 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6599298715591431 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06547678261995316 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9980829954147339 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11256860941648483 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9980829954147339 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9833024740219116 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12352090328931808 *jet/accuracy_1_of_2‚0.7200754880905151 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09923403710126877 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08602386713027954 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4460511803627014 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1571643054485321 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07970461994409561 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3437909185886383 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9358974695205688 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5649443864822388 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5204806327819824 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5403302311897278 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5121389031410217 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 particle/accuracy_1_of_2‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3589743375778198 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6654825806617737 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5002347826957703 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6719309687614441 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9904336929321289 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10283957421779633 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9358974695205688 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5782051682472229 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.44312766194343567 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7350120544433594 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09588345140218735 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5503978729248047 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8093390464782715 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5028634071350098 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9935956597328186 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6089091897010803 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9921373128890991 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/2FRt01ì„DL'SRqqt*FBt/event_purity‚0.02290360815823078 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.26380592584609985 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.431854248046875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5097834467887878 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5688504576683044 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2846153974533081 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13415423035621643 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0694444477558136 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7280586957931519 ,jet/accuracy_4_of_4‚0.008503401651978493 trainer/global_step‚1679 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895554184913635 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09931546449661255 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09507758915424347 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9236111640930176 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.22528694570064545 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6812203526496887 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9839401841163635 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09154290705919266 & Purity/*FRt1SRqqt2FBt/FRt_purity‚1 _step‚137 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16071844100952148 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7280586957931519 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.314706027507782 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2152976542711258 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10361064225435257 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5887454748153687 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9929472804069519 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08311386406421661 (particle/accuracy‚0.8324381113052368 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8651718497276306 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893869161605835 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9936696887016296 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08311386406421661 +jet/accuracy_3_of_4‚0.08725793659687042 particle/accuracy_1_of_3‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6818182468414307 _runtime‚3066.675746706 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5526687502861023 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9968253970146179 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.3214285671710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.806671142578125 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3116244375705719 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5994558930397034 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3214285671710968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4789951741695404 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12708796560764313 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462454557418823 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9062679409980774 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17700064182281494 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0694444477558136 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 +jet/accuracy_2_of_2‚0.10641621798276901 +jet/accuracy_3_of_3‚0.04157452657818794 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.678127646446228 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9803343415260315 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.09742388874292374 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02082418091595173 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9834275841712952 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4349741041660309 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9901466965675354 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16354084014892578 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5722946524620056 particle/accuracy_3_of_4‚1 *loss/val_total_loss‚13.992018339312702 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4490644633769989 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991411566734314 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5292234420776367 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4979889392852783 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9943712949752808 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08873860538005829 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 $ _timestamp‚1.784842303317657e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 *jet/accuracy_1_of_3‚0.8832831978797913 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5963945984840393 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7389612793922424 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5533338785171509 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6953664422035217 7validation_average_jet_accuracy‚0.29154303669929504 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_2_of_4‚0.717983067035675 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11158867180347443 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5088325142860413 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8748293519020081 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.19688645005226135 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08976191282272339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3958333432674408 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6695500612258911 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862300157546997 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11386571079492569 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9777796864509583 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.0997118353843689 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 epoch‚5 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42868539690971375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.944444477558136 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1769939810037613 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.5901012420654297 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 *jet/accuracy_2_of_3‚0.4067316949367523 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0888419896364212 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5279358625411987 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5296761393547058 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0972597748041153 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0018315018387511373 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9823979735374451 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1281767338514328 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011300464160740376 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7327653765678406 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0009920635493472219 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9647959470748901 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3706994950771332 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5437353253364563 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9928392767906189 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 +particle/sensitivity‚0.5621946454048157 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7458465695381165 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9583333730697632 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4848484694957733 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.37690314650535583 /particle/accuracy_3_of_3‚0.9985468983650208üN"<ß œx 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5533338785171509 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6953664422035217 7validation_average_jet_accuracy‚0.29154303669929504 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_2_of_4‚0.717983067035675 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11158867180347443 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5088325142860413 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8748293519020081 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.19688645005226135 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08976191282272339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3958333432674408 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6695500612258911 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862300157546997 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11386571079492569 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9777796864509583 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.0997118353843689 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 epoch‚5 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42868539690971375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.944444477558136 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1769939810037613 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.5901012420654297 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 *jet/accuracy_2_of_3‚0.4067316949367523 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0888419896364212 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5279358625411987 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5296761393547058 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0972597748041153 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0018315018387511373 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9823979735374451 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1281767338514328 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011300464160740376 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7327653765678406 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0009920635493472219 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9647959470748901 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3706994950771332 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5437353253364563 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9928392767906189 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 +particle/sensitivity‚0.5621946454048157 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7458465695381165 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9583333730697632 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4848484694957733 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.37690314650535583 /particle/accuracy_3_of_3‚0.9985468983650208 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 *jet/accuracy_1_of_4‚0.9399362802505493 particle/accuracy_1_of_4‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026101233437657356 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6883958578109741 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9583333730697632 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11158985644578934 * Purity/2FRt0SRqqt2FBt/FRt_purity‚0.625 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.22094236314296722 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47725802659988403 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09154290705919266 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08353257179260254 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3706302046775818 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9924100041389465 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7340009808540344 +jet/accuracy_1_of_1‚0.24097482860088348 /particle/accuracy_4_of_4‚0.8886705636978149 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6818182468414307 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6724880933761597 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5403302311897278 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9933681488037109 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9883203506469727 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07533912360668182 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6599298715591431 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06547678261995316 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9980829954147339 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11256860941648483 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9980829954147339 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9833024740219116 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12352090328931808 *jet/accuracy_1_of_2‚0.7200754880905151 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09923403710126877 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08602386713027954 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4460511803627014 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1571643054485321 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07970461994409561 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3437909185886383 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9358974695205688 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5649443864822388 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5204806327819824 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5403302311897278 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5121389031410217 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 particle/accuracy_1_of_2‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3589743375778198 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6654825806617737 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5002347826957703 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6719309687614441 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9904336929321289 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10283957421779633 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9358974695205688 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5782051682472229 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.44312766194343567 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7350120544433594 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09588345140218735 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5503978729248047 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8093390464782715 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5028634071350098 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9935956597328186 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6089091897010803 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9921373128890991 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02290360815823078 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.26380592584609985 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.431854248046875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5097834467887878 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5688504576683044 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2846153974533081 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13415423035621643 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0694444477558136 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7280586957931519 ,jet/accuracy_4_of_4‚0.008503401651978493 trainer/global_step‚1679 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895554184913635 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09931546449661255 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09507758915424347 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9236111640930176 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.22528694570064545 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6812203526496887 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9839401841163635 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09154290705919266 & Purity/*FRt1SRqqt2FBt/FRt_purity‚1 _step‚137 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16071844100952148 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7280586957931519 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.314706027507782 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2152976542711258 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10361064225435257 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5887454748153687 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9929472804069519 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08311386406421661 (particle/accuracy‚0.8324381113052368 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8651718497276306 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893869161605835 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9936696887016296 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08311386406421661 +jet/accuracy_3_of_4‚0.08725793659687042 particle/accuracy_1_of_3‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6818182468414307 _runtime‚3066.675746706 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5526687502861023 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9968253970146179 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.3214285671710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.806671142578125 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3116244375705719 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5994558930397034 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3214285671710968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4789951741695404 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12708796560764313 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462454557418823 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9062679409980774 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17700064182281494 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0694444477558136 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 +jet/accuracy_2_of_2‚0.10641621798276901 +jet/accuracy_3_of_3‚0.04157452657818794 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.678127646446228 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9803343415260315 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.09742388874292374 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02082418091595173 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9834275841712952 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4349741041660309 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9901466965675354 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16354084014892578 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5722946524620056 particle/accuracy_3_of_4‚1 *loss/val_total_loss‚13.992018339312702 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4490644633769989 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991411566734314 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5292234420776367 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4979889392852783 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9943712949752808 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08873860538005829 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 $ _timestamp‚1.784842303317657e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 *jet/accuracy_1_of_3‚0.8832831978797913 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5963945984840393 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7389612793922424 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0‰µ6ÁeFà ‚(:»  ÇŠÓöƒ‹´ disk.rbd1.out‚ 13.888671875 disk.drbd1226.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51323213450893 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.dm-0.in‚ 51.5234375 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 network.recv‚226104 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 85.3359375 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 85.3359375 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0# memory_percent‚85.0121021270752% disk./.usageGB‚1260.8482055664062 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.02734375# disk.dm-0.out‚10765.81982421875 disk.drbd1965.out‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.742798487345379 disk.drbd1356.out‚0& disk.nvme0n1.out‚10765.81982421875# proc.memory.rssMB‚ 2885.91015625 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 cpu‚25.266869929810202 network.sent‚ 2.968497e+06 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.02734375) proc.memory.availableMB‚ 3683.42578125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 51.5234375 disk.drbd1817.in‚0¸”(Ÿá ‚(:” ÇŠÓÓñé³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚302.134' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.71133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cr¯>Fâ ‚(:»  ÖŠÓ™¢‡´ network.recv‚226254 disk.drbd1673.in‚0 disk.dm-0.in‚ 51.5234375 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.02734375* proc.memory.percent‚11.742798487345379 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0# memory_percent‚85.0121021270752# disk./.usageGB‚1260.84818649292 disk.nvme0n1.in‚ 51.5234375 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 13.951171875 disk.drbd1584.in‚0) proc.memory.availableMB‚ 3683.42578125# proc.memory.rssMB‚ 2885.91015625 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.51323106781973 disk.sda.out‚ 85.56640625 disk.sda2.out‚ 85.56640625 disk.drbd1931.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.02734375 disk.drbd1828.out‚0 network.sent‚ 2.968647e+06 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 cpu‚25.266949487106913 disk.drbd1510.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0& disk.nvme0n1.out‚10765.96044921875# disk.dm-0.out‚10765.96044921875 disk.drbd1828.in‚0Qd~ Ÿã ‚(:” ÖŠÓÚ¦è³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚149.212' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚49.737333333333325 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[˜(Ôžä ‚(:“ åŠÓŒ…Ö³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚277.898' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.63266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š#K—?å ‚(:´  åŠÓå ö³ disk.drbd1938.in‚0 network.recv‚229239 disk.sda.out‚ 86.23046875 disk.drbd1965.out‚0 disk.drbd1584.in‚0" disk.nvme0n1.out‚ 11735.8671875 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2885.91015625 disk.sda2.in‚ 0.02734375 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 51.5234375 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 cpu‚25.266870684843376 disk.drbd1289.in‚0 network.sent‚ 3.028312e+06 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 86.23046875 disk.drbd1285.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8443870544434 disk.drbd1226.out‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51301858333024 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 51.5234375 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0' proc.memory.availableMB‚ 3683.421875 disk.drbd1938.out‚0 disk.dm-0.out‚ 11735.8671875 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 13.951171875* proc.memory.percent‚11.742798487345379 disk.sda.in‚ 0.02734375 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0$ memory_percent‚85.01211802164713]MvV2æ - _wandbruntime‚3105  _runtime‚3105˳/vÉç à ' lr-AdamW/pg1‚0.00011406468528928418 ' lr-AdamW/pg2‚0.00011406468528928418 trainer/global_step‚1699 % _timestamp‚1.7848423417111616e+09 _runtime‚3105.068385577 _step‚138úÅbÎè È ' lr-AdamW/pg1‚0.00011406468528928418 ' lr-AdamW/pg2‚0.00011406468528928418 trainer/global_step‚1699 % _timestamp‚1.7848423417111616e+09 _runtime‚3105.068385577 _step‚138Š¸áûí2é - _wandbruntime‚3105  _runtime‚3105ß„A•ê êO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14764906 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134082382414848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚491175788686998 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚489403815476272 ]IDe†aX—[%viceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚489597938329600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚1699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491372883060224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15792937 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4534060 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491371511450624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134082014323712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚489403790048304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30557843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15792937 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚489597938329600 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1770628599440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚489597912156160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 408872448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491371511450624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1774944730624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5184935 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136253310730752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2170887534592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30547773 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2170928315904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14764900 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14764900 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚491174418647744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4534055 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136252901858304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚40781312 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1773599294464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _step‚139 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14764906 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5185236 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚489597912156160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491372883060224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1771973210726 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9719296 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9718990 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15782873 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1773599294464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 $ _timestamp‚1.784842341712212e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15782873 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3105.069708618 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6¹ r,õ'ë ïO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1773599294464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _step‚139 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14764906 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5185236 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚489597912156160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491372883060224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1771973210726 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9719296 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9718990 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15782873 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1773599294464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 $ _timestamp‚1.784842341712212e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15782873 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3105.069708618 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14764906 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134082382414848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚491175788686998 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚489403815476272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚489597938329600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚1699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491372883060224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15792937 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4534060 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491371511450624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134082014323712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚489403790048304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30557843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15792937 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚489597938329600 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1770628599440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚489597912156160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 408872448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491371511450624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1774944730624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5184935 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136253310730752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2170887534592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30547773 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2170928315904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14764900 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14764900 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚491174418647744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4534055 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136252901858304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚40781312 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755‹Tdð2ì - _wandbruntime‚3106  _runtime‚3106.=”-û&í õM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚489682617250048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724602 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2172156007424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14773273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5188212 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491652814865920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1774621802496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1774621802496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134160600689664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚310 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30575306 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30575306 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚489682591822080 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚489876821446656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1775967245824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1775967245824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136332756697088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4536695 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4536700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚305 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134160968780800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491651443249152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚489876847620096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30565222 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14773273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491652814865920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15802033 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14773267 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15802033 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚1699 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9724912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _step‚140 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30565222 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42871296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 % _timestamp‚1.7848423431903872e+09 _runtime‚3106.547997122 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15791955 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚491455611262526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2172198878720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚489876821446656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724602 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2172156007424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14773273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5188212 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491652814865920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1774621802496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1774621802496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134160600689664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚310 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30575306 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30575306 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚489682591822080 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚489876821446656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1775967245824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1775967245824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136332756697088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4536695 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4536700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚305 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134160968780800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491651443249152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚489876847620096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30565222 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14773273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491652814865920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15802033 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14773267 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15802033 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚1699 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9724912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _step‚140 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30565222 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42871296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 % _timestamp‚1.7848423431903872e+09 _runtime‚3106.547997122 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15791955 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚491455611262526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2172198878720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚489876821446656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 Fó ‚(:³  ôŠÓí¿‚´ disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 network.recv‚229347 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2885.91015625* proc.memory.percent‚11.742798487345379) disk./.usagePercent‚70.51301751664103 disk.sda.out‚ 86.45703125 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 network.sent‚ 3.02842e+06 disk.sda2.out‚ 86.45703125 disk.dm-0.in‚ 51.5234375 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.02734375 disk.sda2.in‚ 0.02734375 disk.drbd2041.in‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 3683.35546875 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0! disk.nvme0n1.out‚ 11736.109375 cpu‚25.266947455537196 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0$ memory_percent‚85.01238822937012 disk.nvme0n1.in‚ 51.5234375 disk.dm-0.out‚ 11736.109375 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 13.9833984375 disk.drbd1510.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0$ disk./.usageGB‚1260.844367980957 disk.sda1.out‚0#]žô ‚(:“ ƒ‘ŠÓ¾Ïç³ gpu.0.gpu‚31.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.917' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.97233333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éª`:Dõ ‚(:¹  ƒ‘ŠÓΉ‹´ disk.drbd1226.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 network.sent‚ 3.028486e+06 disk.drbd2078.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51304589057386! disk.dm-0.out‚11736.462890625 disk.rbd1.out‚ 14.0771484375 disk.drbd1550.in‚0) proc.memory.availableMB‚ 3683.21484375 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda.in‚ 0.02734375% disk./.usageGB‚1260.8448753356934 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 86.7734375* proc.memory.percent‚11.742798487345379 network.recv‚229413 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1673.in‚0 cpu‚25.266944952559168 disk.drbd1510.out‚0 disk.dm-0.in‚ 51.5234375 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 51.5234375 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 86.7734375$ memory_percent‚85.01296043395996# proc.memory.rssMB‚ 2885.91015625 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0$ disk.nvme0n1.out‚11736.462890625 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.02734375 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0ÞYšžö ‚(:“ ’‘ŠÓÅíÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚296.516' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.83866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0){ã;÷ ‚(:°  ’‘ŠÓ¥¬û³ disk.drbd2111.in‚0$ memory_percent‚85.01486778259277 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51304717060091$ disk./.usageGB‚1260.844898223877 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚11736.748046875 disk.sda.out‚ 87.1015625 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.sda.in‚ 0.02734375 cpu‚25.266867164454766 disk.drbd2041.out‚0 disk.nvme0n1.in‚52.625 disk.sda2.out‚ 87.1015625 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.dm-0.in‚52.625* proc.memory.percent‚11.742798487345379 network.recv‚230386 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2885.91015625 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1552.out‚0! disk.dm-0.out‚11736.748046875 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0) proc.memory.availableMB‚ 3682.74609375 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 network.sent‚ 3.034319e+06 disk.rbd1.out‚ 14.1396484375 disk.drbd1285.out‚0 disk.sda2.in‚ 0.02734375|õSŸø ‚(:” ¡‘ŠÓ™¼Û³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚310.208' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.40266666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô‚Jù ‚(:¿  ¡‘ŠÓæ­ý³ disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0% disk./.usageGB‚1260.8448600769043 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2885.91015625 disk.drbd1552.in‚0 disk.sda2.out‚ 87.51953125 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 network.recv‚230452 disk.drbd1285.in‚0 disk.drbd1817.in‚0# disk.dm-0.out‚11737.05029296875 disk.drbd1828.out‚0 disk.rbd1.out‚ 14.1396484375 disk.drbd1584.in‚0 cpu‚25.267022313181972( proc.memory.availableMB‚ 4578.1484375 disk.sda.out‚ 87.51953125( disk./.usagePercent‚70.5130450372225 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 58.47265625 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.sda.in‚ 0.02734375 proc.cpu.threads‚17 network.sent‚ 3.034385e+06 disk.drbd1289.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚11737.05029296875 disk.drbd1510.out‚0 disk.dm-0.in‚ 58.47265625 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.742798487345379 disk.drbd1550.out‚0 disk.sda2.in‚ 0.02734375$ memory_percent‚81.371466318766271”Óžú ‚(:“ °‘ŠÓߧݳ gpu.0.gpu‚52.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.946' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.64866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜU˜gAû ‚(:¶  °‘ŠÓÝšÿ³ network.recv‚230560 disk.sda2.in‚ 0.02734375 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda.out‚ 87.7734375) proc.memory.percent‚11.73876126607259 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0( proc.memory.availableMB‚ 4314.3359375 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 87.7734375 disk.drbd2078.out‚0 cpu‚25.26717600218507 disk.drbd1584.out‚0% disk./.usageGB‚1260.8448867797852" disk.dm-0.out‚11737.2548828125 disk.drbd2111.out‚0 disk.rbd1.out‚ 14.171875 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.02734375 disk.drbd2118.out‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51304653058739 disk.drbd1931.out‚0% disk.nvme0n1.out‚11737.2548828125 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2884.91796875 disk.dm-0.in‚ 58.47265625$ memory_percent‚82.44492212931314 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1552.out‚0 network.sent‚ 3.034493e+06 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 58.47265625Ȧ%¾2ü - _wandbruntime‚3192  _runtime‚3192pïÔ¦Éý à _runtime‚3192.224378745 _step‚142 ' lr-AdamW/pg1‚0.00011384700529901181 ' lr-AdamW/pg2‚0.00011384700529901181 trainer/global_step‚1749 % _timestamp‚1.7848424288670979e+09ÆÎÞBÎþ È _step‚142 ' lr-AdamW/pg1‚0.00011384700529901181 ' lr-AdamW/pg2‚0.00011384700529901181 trainer/global_step‚1749 % _timestamp‚1.7848424288670979e+09 _runtime‚3192.224378745Ž¼Ñ¼ô2ÿ - _wandbruntime‚3192  _runtime‚3192t‡H5ñ'€ ëO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4666055 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15183256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1821732482248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503543376680960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31422075 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505368165531648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826134286848 _runtime‚3192.228952763 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16248889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16238825 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140245410169344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503543402854400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5333037 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚503343878737104 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31432145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2233709456896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1824788850688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505369537141248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848424288682554e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503543376680960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1824788850688 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9999097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505369537141248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9998755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4666055 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15183256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1821732482248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503543376680960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_p{ °ool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31422075 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505368165531648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826134286848 _runtime‚3192.228952763 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16248889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16238825 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140245410169344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503543402854400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5333037 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚503343878737104 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31432145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2233709456896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1824788850688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505369537141248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848424288682554e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503543376680960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1824788850688 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9999097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505369537141248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9998755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848424303479447e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚503622680510880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505648097330176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2234905226240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _step‚144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚344 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5335660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15191623 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16247907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31449608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503822312144896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1825811358720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4668695 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1824097895286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31449608 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503822312144896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505649468946944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138089918987264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31439524 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚1749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15191623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503822285971456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827156802048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1825811358720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15191617 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10004699 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16257985 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5335999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10004355 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚505445433794768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚503822285971456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15191617 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚503622705938848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15191617 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚503622705938848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848424303479447e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚503622680510880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505648097330176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2234905226240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _step‚144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚344 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5335660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15191623 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16247907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31449608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503822312144896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1825811358720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4668695 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1824097895286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31449608 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503822312144896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505649468946944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138089918987264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31439524 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚1749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15191623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503822285971456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827156802048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1825811358720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15191617 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10004699 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16257985 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5335999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10004355 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚8248–Â2… - _wandbruntime‚3193  _runtime‚3193>“8ˆ € 2loss/SRqqt2/assignment_loss‚0.6484849502539541 &loss/total_loss‚12.474836864462697 0loss/FRt1/assignment_loss‚2.7106199008562872 2loss/SRqqt1/assignment_loss‚0.8717674927325332 0loss/FBt2/assignment_loss‚0.1503189599742552 /loss/FRt1/detection_loss‚0.6026910589353065 1loss/SRqqt1/detection_loss‚1.2870450825958522 1loss/SRqqt2/detection_loss‚0.7300451068011446 /loss/FBt1/detection_loss‚1.1260040918434109 /loss/FBt2/detection_loss‚0.8340352452027676 0loss/FRt2/assignment_loss‚1.4295592051396668 0loss/FBt1/assignment_loss‚0.5532700715207115 epoch‚6 % _timestamp‚1.7848424303816874e+09 _runtime‚3193.738931421 _step‚145 /loss/FRt2/detection_loss‚1.5309956986068067 trainer/global_step‚1749‚¾i ‡ … 2loss/SRqqt2/assignment_loss‚0.6484849502539541 &loss/total_loss‚12.474836864462697 0loss/FRt1/assignment_loss‚2.7106199008562872 2loss/SRqqt1/assignment_loss‚0.8717674927325332 0loss/FBt2/assignment_loss‚0.1503189599742552 /loss/FRt1/detection_loss‚0.6026910589353065 1loss/SRqqt1/detection_loss‚1.2870450825958522 1loss/SRqqt2/detection_loss‚0.7300451068011446 /loss/FBt1/detection_loss‚1.1260040918434109 /loss/FBt2/detection_loss‚0.8340352452027676 0loss/FRt2/assignment_loss‚1.4295592051396668 0loss/FBt1/assignment_loss‚0.5532700715207115 epoch‚6 % _timestamp‚1.7848424303816874e+09 _runtime‚3193.738931421 _step‚145 /loss/FRt2/detection_loss‚1.5309956986068067 trainer/global_step‚1749‘Ed«žˆ ‚(:“ ¿‘ŠÓ³ÒÛ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.584' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.86133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,æŠ7>‰ ‚(:³  ¿‘ŠÓ§Íú³ disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.02734375 disk.nvme0n1.in‚ 58.47265625 disk.drbd1510.out‚0 disk.sda2.out‚ 88.046875 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 58.47265625" disk.dm-0.out‚11737.7626953125) proc.memory.percent‚11.73876126607259 disk.drbd2078.in‚0 disk.sda.out‚ 88.046875 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51305293072262 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0% disk.nvme0n1.out‚11737.7626953125# proc.memory.rssMB‚ 2884.91796875 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 cpu‚25.26717417027957 proc.cpu.threads‚17% disk./.usageGB‚1260.8450012207031 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.265625 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.02734375$ memory_percent‚83.55474472045898 network.sent‚ 3.09969e+06 disk.drbd1938.in‚0 disk.drbd1828.in‚0( proc.memory.availableMB‚ 4041.5859375 disk.drbd1673.out‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 network.recv‚232510Æ ^žŠ ‚(:“ ΑŠÓ²‹Á³ gpu.0.gpu‚64.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.77' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.92333333333332 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëåÚ‹ ‚(:¸  ΑŠÓòÔã³" disk.dm-0.out‚11738.4267578125 disk.drbd2078.in‚0 disk.drbd2111.out‚0 network.sent‚ 3.099798e+06 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51305293072262 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.02734375 disk.drbd1931.in‚0 disk.drbd1938.out‚0 cpu‚25.267328905639157 disk.drbd1938.in‚0) proc.memory.availableMB‚ 3796.99609375 network.recv‚232618 disk.drbd1289.in‚0 disk.dm-0.in‚ 58.47265625 disk.drbd1828.out‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17) proc.memory.percent‚11.73876126607259 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.02734375 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.sda2.out‚ 88.3828125 disk.rbd1.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚11738.4267578125 disk.drbd2118.in‚0# proc.memory.rssMB‚ 2884.91796875 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0% disk./.usageGB‚1260.8450012207031 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 14.328125 disk.nvme0n1.in‚ 58.47265625 disk.drbd1510.out‚0$ memory_percent‚84.54998334248862 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 dis@dÐ.k.drbd1673.in‚0 disk.sda.out‚ 88.3828125Ö Ä„ŸŒ ‚(:” Ý‘ŠÓê麳 gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚312.902' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.30066666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E¢|ãC ‚(:¸  Ý‘ŠÓ¯ Ý³ disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 cpu‚25.26724996517806 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚11739.05322265625 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2884.91796875 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.02734375$ memory_percent‚85.76448758443198% disk./.usageGB‚1260.8449058532715 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 88.8515625 disk.sda2.out‚ 88.8515625( disk./.usagePercent‚70.5130475972766 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 58.47265625# disk.dm-0.out‚11739.05322265625 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 3498.51953125 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 58.47265625 network.sent‚ 3.099864e+06 disk.drbd2111.in‚0 disk.rbd1.out‚ 14.328125 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.02734375 disk.drbd1356.out‚0 disk.drbd1289.in‚0) proc.memory.percent‚11.73876126607259 network.recv‚232684¼XeæGŽ ‚(:¼  ì‘ŠÓ¾†´ disk.sda2.out‚ 89.33203125 disk.drbd1289.in‚0& proc.memory.availableMB‚ 3201.71875* proc.memory.percent‚11.740223566691082 network.sent‚ 3.108066e+06 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0& disk.nvme0n1.out‚11739.44677734375 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 58.47265625 disk.sda.in‚ 0.02734375 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 network.recv‚233922 disk.sda2.in‚ 0.02734375 disk.dm-0.in‚ 58.47265625 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1673.in‚0 cpu‚25.26724209459247 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚11739.44677734375 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0$ memory_percent‚86.97217305501303) disk./.usagePercent‚70.51305037066852$ disk./.usageGB‚1260.844955444336 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.3603515625 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 89.33203125 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17uz–Ùž ‚(:“ ì‘ŠÓ¢“ç³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.802' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.60066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A÷•dž ‚(:“ û‘ŠÓÂø²³ gpu.0.gpu‚42.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.618' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.87266666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ô@‘ ‚(:µ  û‘ŠÓ’Æϳ disk.drbd2041.out‚0 disk.rbd1.out‚ 14.4541015625$ memory_percent‚88.13047409057617 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 2917.0546875 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.02734375 disk.drbd1289.out‚0 disk.dm-0.out‚ 11740.296875 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd2041.in‚0! disk.nvme0n1.out‚ 11740.296875 disk.drbd1226.out‚0 disk.sda2.out‚ 89.72265625 disk.drbd1931.out‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 58.47265625 cpu‚25.267474434625473 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 58.47265625 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 89.72265625 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 network.sent‚ 3.108174e+06 network.recv‚234030) disk./.usagePercent‚70.51305037066852 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.740223566691082$ disk./.usageGB‚1260.844955444336 disk.sda.in‚ 0.02734375 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0<%((ž’ ‚(:“ Š’ŠÓÿãÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚264.179' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.05966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ÙoŽL“ ‚(:Á  Š’ŠÓ³Ñù³# proc.memory.rssMB‚ 2885.27734375 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚11740.73583984375 disk.sda.in‚ 0.02734375 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.8449363708496 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 58.47265625# disk.dm-0.out‚11740.73583984375) disk./.usagePercent‚70.51304930397933 disk.drbd1931.in‚0 disk.drbd1550.in‚0 network.sent‚ 3.173447e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 90.04296875 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0$ memory_percent‚88.96886507670084) proc.memory.availableMB‚ 2711.01171875 proc.cpu.threads‚17 disk.sda2.in‚ 0.02734375 disk.nvme0n1.in‚ 58.47265625 disk.drbd1584.out‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.740223566691082 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 14.5166015625 cpu‚25.267314107778308 disk.drbd2118.out‚0 disk.drbd1828.out‚0 network.recv‚236405 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 90.04296875 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0Q’Ã2” - _wandbruntime‚3279  _runtime‚3279è>…È•  ' lr-AdamW/pg1‚0.00011362207904038965 ' lr-AdamW/pg2‚0.00011362207904038965 trainer/global_step‚1799 $ _timestamp‚1.784842515809122e+09 _runtime‚3279.166359822 _step‚146­ÁXƒÍ– Ç $ _timestamp‚1.784842515809122e+09 _runtime‚3279.166359822 _step‚146 ' lr-AdamW/pg1‚0.00011362207904038965 ' lr-AdamW/pg2‚0.00011362207904038965 trainer/global_step‚1799’%åZ^2— - _wandbruntime‚3279  _runtime‚3279'b(uó'˜ íO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16704841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141941019010048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚391 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872836365056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1875978406912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15601600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519366191222272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15601600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480306 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32296377 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517283967425904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15601606 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7848425158101563e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16704841 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 trainer/global_step‚1799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1877323843072 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5479920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519364819612672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10277975 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1874180976342 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4798060 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141940650918912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517283992853872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144237130485248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 436135424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2296111475200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚519158173830214 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2296043430912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517488867379200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517488841205760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144236694349824 _runtime‚3279.170949241 _step‚147 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070²‰²sø'™ òO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517488841205760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144236694349824 _runtime‚3279.170949241 _step‚147 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16704841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141941019010048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚391 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872836365056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1875978406912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15601600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519366191222272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15601600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480306 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32296377 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517283967425904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15601606 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7848425158101563e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16704841 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 trainer/global_step‚1799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1877323843072 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5479920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519364819612672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10277975 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1874180976342 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4798060 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141940650918912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517283992853872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144237130485248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 436135424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2296111475200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚519158173830214 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2296043430912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517488867379200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0“×ûj2š - _wandbruntime‚3280  _runtime‚3280:ùïk¶ › ÷M UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1877000914944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16713937 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16703859 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16713937 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1873857166696 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517767776669696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15609973 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚1799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1878346358272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519646123027968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2297349880320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15609967 EÑÒ “G?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1875201778094 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4800700 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4800695 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848425172866166e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519646123027968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519644751411200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1878346358272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚517562794627648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚148 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚3280.644208213 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517767750496256 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32313826 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10283510 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142019605376000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚519437996405742 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2297288134656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32323910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚517767776669696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32313826 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5483184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517767776669696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15609973 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚1799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1878346358272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519646123027968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2297349880320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15609967 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1875201778094 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4800700 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4800695 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848425172866166e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519646123027968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519644751411200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1878346358272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚517562794627648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚148 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚3280.644208213 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517767750496256 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32313826 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10283510 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142019605376000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚519437996405742 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2297288134656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32323910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚517767776669696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32313826 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5483184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EyI§ ‚(:¾  Æ’ŠÓý…ÿ³) proc.memory.availableMB‚ 2710.41015625 network.recv‚239237 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.sda2.out‚ 91.6796875* proc.memory.percent‚11.740223566691082 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 58.47265625 disk.sda1.out‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0$ memory_percent‚88.97131284077963 disk.drbd2078.out‚0 disk.dm-0.in‚ 58.47265625 disk.drbd2041.in‚0& disk.nvme0n1.out‚11742.23876953125 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.02734375# proc.memory.rssMB‚ 2885.27734375) disk./.usagePercent‚70.51304951731716# disk.dm-0.out‚11742.23876953125 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 cpu‚25.267450563818027 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0% disk./.usageGB‚1260.8449401855469 disk.drbd1356.out‚0 disk.rbd1.out‚ 14.705078125 disk.sda2.in‚ 0.02734375 disk.drbd1356.in‚0 disk.sda.out‚ 91.6796875 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 network.sent‚ 3.239063e+06 disk.drbd2078.in‚0~ïçw”¨ ‚(:‰ Õ’ŠÓĽµ³ gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚310.017' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚103.339 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÄñ*H© ‚(:½  Õ’ŠÓ™‹Ø³ disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 cpu‚25.267525964026973 proc.cpu.threads‚17) disk./.usagePercent‚70.51304951731716 disk.drbd1226.in‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8449401855469 disk.drbd1931.in‚0 disk.dm-0.in‚ 58.47265625 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 58.47265625 disk.drbd1965.in‚0 disk.drbd1828.out‚0# memory_percent‚88.9716943105062 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.sda.out‚ 92.0390625 disk.sda2.in‚ 0.02734375 network.recv‚239345 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 14.705078125 disk.drbd1673.out‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.740223566691082 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 2710.31640625 network.sent‚ 3.239171e+06 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 92.0390625 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.02734375# disk.dm-0.out‚11742.53076171875 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚11742.53076171875# proc.memory.rssMB‚ 2885.27734375 disk.sda1.in‚0 disk.sda1.out‚0&𞪠‚(:“ ä’ŠÓ‹çå³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.174' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.39133333333332 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼“gI« ‚(:¾  ä’ŠÓò…´$ memory_percent‚88.97082010904947& proc.memory.availableMB‚ 2710.53125* proc.memory.percent‚11.740223566691082 disk.drbd2118.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51305058400636& disk.nvme0n1.out‚11742.67919921875 disk.drbd1584.out‚0 disk.sda2.out‚ 92.39453125# proc.memory.rssMB‚ 2885.27734375 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.02734375 disk.drbd1510.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.dm-0.in‚ 58.47265625 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.9169921875 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 58.47265625 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 92.39453125 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 network.sent‚ 3.244879e+06 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.sda2.in‚ 0.02734375% disk./.usageGB‚1260.8449592590332 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0# disk.dm-0.out‚11742.67919921875 cpu‚25.267516463517307 network.recv‚240532 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0Lž»2¬ - _wandbruntime‚3366  _runtime‚3366vuÒÉ­ à ' lr-AdamW/pg1‚0.00011338993599609692 ' lr-AdamW/pg2‚0.00011338993599609692 trainer/global_step‚1849 % _timestamp‚1.7848426026593952e+09 _runtime‚3366.016642616 _step‚150Q/Uή È ' lr-AdamW/pg1‚0.00011338993599609692 ' lr-AdamW/pg2‚0.00011338993599609692 trainer/global_step‚1849 % _timestamp‚1.7848426026593952e+09 _runtime‚3366.016642616 _step‚150–e;ñ2¯ - _wandbruntime‚3366  _runtime‚3366ãhG¾˜° ìO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1928513399296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚533149366401822 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531224056114704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 $ _timestamp‚1.784842602660509e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533361473693696 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17160793 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927167963136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16019956 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1406294528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531224081542672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531434331904000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17150729 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_sËnÐÚZ tart/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148228705886208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5626940 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2358368578560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33170679 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5627322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2358306825728 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533361473693696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚533362845303296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928513399296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚1849 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17150729 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533362845303296 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16019950 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17160793 _runtime‚3366.021942966 _step‚151 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4930060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531434305730560 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10557382 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10557382 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5626940 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2358368578560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33170679 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5627322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2358306825728 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533361473693696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚533362845303296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928513399296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚1849 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17150729 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533362845303296 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16019950 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254—J”2² - _wandbruntime‚3367  _runtime‚3367Nyý&³ ÷M NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16028323 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚533642777108992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928190471168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16028323 % _timestamp‚1.7848426041362941e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚366 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33198212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533641405492224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16028317 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17169889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148308102999552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531713241194496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145948923673600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2359547417088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1924961049504 trainer/global_step‚1849 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16028317 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17159811 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10562545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33188128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5630216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33198212 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17169889 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17159811 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531502883316448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10562916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚371 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56Õ©’c]´ üM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4932700 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚533429188977350 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚533642777108992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531713215021056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16028323 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚533642777108992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928190471168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16028323 % _timestamp‚1.7848426041362941e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚366 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33198212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533641405492224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16028317 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17169889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148308102999552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531713241194496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145948923673600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2359547417088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1924961049504 trainer/global_step‚1849 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16028317 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17159811 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10562545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33188128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5630216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33198212 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17169889 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17159811 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531502883316448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10562916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚371 E ‚(:³  ¾“ŠÓ”šÚ³ disk.sda2.in‚ 0.02734375# disk.dm-0.out‚11744.78857421875 disk.drbd1552.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.740223566691082 disk.drbd2118.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 network.recv‚246802 disk.drbd1226.in‚0 disk.drbd1817.out‚0 network.sent‚ 3.37879e+06 disk.drbd1673.out‚0" memory_percent‚88.970947265625 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1510.in‚0 disk.dm-0.in‚ 58.47265625 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 58.47265625$ disk./.usageGB‚1260.846363067627 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0" proc.memory.availableMB‚2710.5 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 15.16796875 disk.drbd1550.in‚0 cpu‚25.267643227192792 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.sda.in‚ 0.02734375 disk.sda.out‚ 94.73828125 disk.sda2.out‚ 94.73828125( disk./.usagePercent‚70.5131290923318 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0& disk.nvme0n1.out‚11744.78857421875 disk.drbd1550.out‚0 disk.drbd1673.in‚0Ì.@Ÿà ‚(:” ¾“ŠÓÇ’½³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.918' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.30600000000001 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö9^”2Ä - _wandbruntime‚3453  _runtime‚3453z,&ÉÅ Ã ' lr-AdamW/pg1‚0.00011315060659476807 ' lr-AdamW/pg2‚0.00011315060659476807 trainer/global_step‚1899 % _timestamp‚1.7848426896605308e+09 _runtime‚3453.018898614 _step‚154ÀLpÎÆ È ' lr-AdamW/pg2‚0.00011315060659476807 trainer/global_step‚1899 % _timestamp‚1.7848426896605308e+09 _runtime‚3453.018898614 _step‚154 ' lr-AdamW/pg1‚0.00011315060659476807š‚NÇÞ2Ç - _wandbruntime‚3453  _runtime‚3453[xŠwò'È ìO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17616745 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545379770255360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5773178 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545164144803504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1975044130672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1978357519360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚547359499384320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16438300 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚547359499384320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16438306 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16438306 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5773550 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547140558973430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979702955520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16438300 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10835233 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚155 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17606681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547139188934176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚547358127774720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149799655605248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5062060 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 431941120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚547358127774720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545379796428800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112ÝþâF É ñO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5062060 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 431941120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚547358127774720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545379796428800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17616745 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545379770255360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5773178 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545164144803504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1975044130672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1978357519360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚547359499384320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16438300 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚547359499384320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16438306 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16438306 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5773550 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547140558973430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979702955520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16438300 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10835233 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚155 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17606681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 344013312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547139188934176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚547358127774720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149799655605248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248›0‰«ß2Ê - _wandbruntime‚3454  _runtime‚34548/hðý&Ë ÷M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 344013312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5064700 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2421603584000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16446667 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149877873880064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚547638059573248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚545658705719296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1979380027392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1899 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚545658679545856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17615763 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16446667 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547420381548958 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10841148 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17615763 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152299477464064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7848426911405256e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34062430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545442972005248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1980725470720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152299903106560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10840774 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34072514 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚547639431190016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545658705719296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2421661135360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1980725470720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547419011509592 _runtime‚3454.498143635 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1977409543710 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34072514 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545658679545856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1979380027392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17625841 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5776079 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545442946577280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17625841 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5776448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚369 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1976064932312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34062430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5064695 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16446667 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547420381548958 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10841148 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17615763 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152299477464064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7848426911405256e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34062430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545442972005248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1980725470720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152299903106560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10840774 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34072514 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚547639431190016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545658705719296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2421661135360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1980725470720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547419011509592 _runtime‚3454.498143635 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1977409543710 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34072514 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545658679545856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1979380027392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17625841 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5776079 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545442946577280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17625841 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5776448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚369 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1976064932312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34062430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5064695 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2421603584000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16446667 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1406294528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149877873880064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚547638059573248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚545658705719296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1979380027392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1899 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚545658679545856œËHÁÅ2Í - _wandbruntime‚3454  _runtime‚3454—M-—Î ÿ 0loss/FRt1/assignment_loss‚2.7723239706444347 0loss/FRt2/assignment_loss‚1.1724323019778269 2loss/SRqqt1/assignment_loss‚0.7567277675812726 1loss/FBt2/assignment_loss‚0.05373881637705973 /loss/FRt1/detection_loss‚0.6303328340405296 %loss/total_loss‚17.91088628158192 epoch‚6 $ _timestamp‚1.784842691175317e+09 2loss/SRqqt2/assignment_loss‚0.6658414190394706 1loss/FBt1/assignment_loss‚0.22071202295802111 /loss/FRt2/detection_loss‚1.5169439124617128 1loss/SRqqt1/detection_loss‚1.2399527427617765 .loss/FBt2/detection_loss‚6.650107998322122 _runtime‚3454.532595083 _step‚157 1loss/SRqqt2/detection_loss‚0.6579402253731468 /loss/FBt1/detection_loss‚1.5738322700445468 trainer/global_step‚1899L'T Ï „ 0loss/FRt1/assignment_loss‚2.7723239706444347 0loss/FRt2/assignment_loss‚1.1724323019778269 2loss/SRqqt1/assignment_loss‚0.7567277675812726 1loss/FBt2/assignment_loss‚0.05373881637705973 /loss/FRt1/detection_loss‚0.6303328340405296 %loss/total_loss‚17.91088628158192 epoch‚6 $ _timestamp‚1.784842691175317e+09 2loss/SRqqt2/assignment_loss‚0.6658414190394706 1loss/FBt1/assignment_loss‚0.22071202295802111 /loss/FRt2/detection_loss‚1.5169439124617128 1loss/SRqqt1/detection_loss‚1.2399527427617765 .loss/FBt2/detection_loss‚6.650107998322122 _runtime‚3454.532595083 _step‚157 1loss/SRqqt2/detection_loss‚0.6579402253731468 /loss/FBt1/detection_loss‚1.5738322700445468 trainer/global_step‚1899ÎÚ¹žÐ ‚(:“ Í“ŠÓʢݳ gpu.0.gpu‚80.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚264.907' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.30233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¿ÒUGÑ ‚(:¼  Í“ŠÓœ–÷³ disk.drbd1226.out‚0 proc.cpu.threads‚17 network.recv‚247872 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 58.47265625 disk.drbd1510.in‚0* proc.memory.percent‚11.740223566691082 disk.drbd1673.out‚0& disk.nvme0n1.out‚11745.02294921875 disk.drbd1817.out‚0 disk.sda.out‚ 95.23828125 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 15.16796875 disk.drbd1552.out‚0) disk./.usagePercent‚70.51314210594009% disk./.usageGB‚1260.8465957641602 disk.drbd1931.in‚0 disk.dm-0.in‚ 58.47265625 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 network.sent‚ 3.387203e+06 disk.sda2.out‚ 95.23828125 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0$ memory_percent‚88.97107442220053& proc.memory.availableMB‚ 2710.46875# proc.memory.rssMB‚ 2885.27734375 disk.drbd1289.out‚0# disk.dm-0.out‚11745.02294921875 disk.drbd1817.in‚0 disk.sda.in‚ 0.02734375 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.sda1.in‚0 cpu‚25.267564144221303 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0¹<TžÒ ‚(:“ Ü“ŠÓ‹çá³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.458' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.15266666666668 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿa±¦ Ó ‚(:µ  Ü“ŠÓ—•„´ disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 15.2001953125 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2885.27734375 proc.cpu.threads‚17$ disk./.usageGB‚1260.846767425537 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0& proc.memory.availableMB‚ 2710.46875 network.sent‚ 3.387269e+06 disk.sda.out‚95.53125 disk.úK4¤3drbd2078.in‚0 disk.drbd2111.out‚0$ memory_percent‚88.97107442220053 network.recv‚247938 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚95.53125 disk.sda1.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0" disk.dm-0.out‚11745.3505859375 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.02734375 disk.drbd1965.in‚0 cpu‚25.267558784249584 disk.drbd1285.out‚0 disk.dm-0.in‚ 58.47265625 disk.sda.in‚ 0.02734375 disk.drbd1285.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.740223566691082 disk.nvme0n1.in‚ 58.47265625 disk.drbd1550.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51315170614293% disk.nvme0n1.out‚11745.3505859375z+šœŸÔ ‚(:” ë“ŠÓûôų gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.108' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.70266666666667 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú³«+CÕ ‚(:¸  ë“ŠÓ¤ùâ³ disk.sda.out‚ 95.8515625" disk.dm-0.out‚11745.5849609375 disk.drbd2041.out‚0% proc.memory.availableMB‚ 2710.4375 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 network.sent‚ 3.387377e+06 disk.sda2.out‚ 95.8515625 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 58.47265625% disk.nvme0n1.out‚11745.5849609375 disk.drbd1584.out‚0 disk.drbd1817.out‚0 network.recv‚248046 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.740223566691082$ memory_percent‚88.97120157877603 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd1938.in‚0 disk.rbd1.out‚ 15.2939453125 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 58.47265625 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.sda.in‚ 0.02734375 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51315063945373% disk./.usageGB‚1260.8467483520508 disk.drbd2078.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 cpu‚25.26770056661463ƒFïžÖ ‚(:“ ú“ŠÓßüº³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚116.744' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.91466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P»õhH× ‚(:½  ú“ŠÓµñÕ³ disk.sda2.in‚ 0.02734375$ memory_percent‚88.97120157877603 disk.drbd1285.in‚0 network.sent‚ 3.452628e+06 disk.drbd2078.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0% proc.memory.availableMB‚ 2710.4375 network.recv‚250215 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.sda.in‚ 0.02734375% disk./.usageGB‚1260.8465423583984 disk.drbd1356.out‚0& disk.nvme0n1.out‚11746.31005859375 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.740223566691082 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 58.47265625 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51313911921032 disk.drbd1828.out‚0 disk.rbd1.out‚ 15.3564453125 disk.drbd1550.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 cpu‚25.267697772086915 disk.drbd1938.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2885.27734375# disk.dm-0.out‚11746.31005859375 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 96.25390625 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 58.47265625 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.sda.out‚ 96.25390625 disk.drbd1673.in‚0®!¹ãžØ ‚(:“ ‰”ŠÓäÝѳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.717' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.90566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i\RÀ:Ù ‚(:¯  ‰”ŠÓÛýé³ disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0% proc.memory.availableMB‚ 2710.4375 proc.cpu.threads‚17 network.recv‚250281 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2885.27734375* proc.memory.percent‚11.740223566691082 disk.dm-0.in‚58.625 disk.drbd1356.out‚0& disk.nvme0n1.out‚11746.45849609375 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8465614318848 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.rbd1.out‚ 15.3564453125 disk.sda.out‚ 96.671875) disk./.usagePercent‚70.51314018589953# disk.dm-0.out‚11746.45849609375 disk.sda2.out‚ 96.671875 disk.drbd2078.out‚0 disk.drbd1828.in‚0 network.sent‚ 3.452694e+06 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.02734375 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0$ memory_percent‚88.97120157877603 disk.drbd1673.in‚0 cpu‚25.267691053062972 disk.nvme0n1.in‚58.625 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0êƒy92Ú - _wandbruntime‚3539  _runtime‚3539Ï ÉÛ Ã ' lr-AdamW/pg1‚0.00011290412220700427 ' lr-AdamW/pg2‚0.00011290412220700427 trainer/global_step‚1949 % _timestamp‚1.7848427763158607e+09 _runtime‚3539.673101364 _step‚158›„e ÎÜ È ' lr-AdamW/pg1‚0.00011290412220700427 ' lr-AdamW/pg2‚0.00011290412220700427 trainer/global_step‚1949 % _timestamp‚1.7848427763158607e+09 _runtime‚3539.673101364 _step‚158žB#2Ý - _wandbruntime‚3539  _runtime‚3539ÀvÖó'Þ íO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153728973902848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚561354781855744 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18062633 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚559104233492304 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11113234 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2482599016960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16856650 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2029547075584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚1949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5194060 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16856656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559325260953600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18072697 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153728605811712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5918806 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561130381505784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18072697 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2030892511744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156211143075840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚561354781855744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16856650 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5919174 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2026148013480 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚561356153465344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559325234780160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559325260953600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚559325234780160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34919283 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2482537264128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3539.674420432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2029547075584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16856656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832q_Æ‚ø'ß òO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34919283 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2482537264128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3539.674420432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2029547075584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16856656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153728973902848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚561354781855744 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18062633 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚559104233492304 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11113234 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2482599016960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16856650 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2029547075584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚1949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5194060 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16856656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559325260953600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18072697 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153728605811712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5918806 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561130381505784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18072697 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2030892511744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156211143075840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚561354781855744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16856650 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5919174 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2026148013480 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚561356153465344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559325234780160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559325260953600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚559325234780160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680Ÿ>óÖøà ‚(:’ ˜”ŠÓ¬Ï³ gpu.0.gpu‚65.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.24' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.74666666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­š99á ‚(:®  ˜”ŠÓÒ´ñ³) disk./.usagePercent‚70.51313783918327 disk.sda2.in‚ 0.02734375 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 15.388671875 disk.sda2.out‚ 97.046875 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.740223566691082 cpu‚25.26775585669309 disk.drbd1673.out‚0 disk.rbd1.in‚0$ memory_percent‚88.97218704223633 disk.drbd1289.in‚0 disk.dm-0.in‚58.625 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 network.recv‚251184 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚58.625 network.sent‚ 3.460856e+06% disk./.usageGB‚1260.8465194702148 disk.drbd2118.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.sda.out‚ 97.046875 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.02734375% disk.nvme0n1.out‚11747.1416015625 disk.drbd1931.in‚0" disk.dm-0.out‚11747.1416015625 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0( proc.memory.availableMB‚ 2710.1953125 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0-‰_2â - _wandbruntime‚3541  _runtime‚3541óïþ¯È ã ÷M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2483772073984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚561636085271040 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11118379 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2483840111104 _step‚160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1949 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34936732 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18081793 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚559604170244096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153807192177664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34946816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2030569583616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16865023 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16865023 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _runtime‚3541.150188807 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34946816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2030569583616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚559383060694048 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚559604144070656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11118755 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7848427777925637e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5196700 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16865017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 436128256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚559604144070656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561410204081200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2027168815120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚561634713654272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚561636085271040 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11118379 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2483840111104 _step‚160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1949 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34936732 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18081793 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚559604170244096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153807192177664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34946816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2030569583616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16865023 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16865023 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _runtime‚3541.150188807 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34946816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2030569583616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚559383060694048 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚559604144070656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11118755 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7848427777925637e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5196700 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16865017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 436128256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚559604144070656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ÖãE2å - _wandbruntime‚3541  _runtime‚3541õiŸ¢æ € /loss/FRt2/assignment_loss‚1.361643129713992 1loss/FBt1/assignment_loss‚0.21735347972400418 1loss/FBt2/assignment_loss‚0.14065528758804535 1loss/SRqqt2/detection_loss‚0.7234697147524923 /loss/FBt2/detection_loss‚1.1046378987661352 epoch‚6 _runtime‚3541.186502261 0loss/FRt1/assignment_loss‚2.6605840128228624 2loss/SRqqt1/assignment_loss‚0.7390122447172274 1loss/SRqqt1/detection_loss‚1.2032971426151593 /loss/FBt1/detection_loss‚0.8952944340074763 &loss/total_loss‚11.960632627005413 trainer/global_step‚1949 % _timestamp‚1.7848427778293204e+09 1loss/SRqqt2/assignment_loss‚0.703775904708667 /loss/FRt1/detection_loss‚0.6230353456571035 /loss/FRt2/detection_loss‚1.5878740319322502 _step‚161Žî© ç … /loss/FRt2/assignment_loss‚1.361643129713992 1loss/FBt1/assignment_loss‚0.21735347972400418 1loss/FBt2/assignment_loss‚0.14065528758804535 1loss/SRqqt2/detection_loss‚0.7234697147524923 /loss/FBt2/detection_loss‚1.1046378987661352 epoch‚6 _runtime‚3541.186502261 0loss/FRt1/assignment_loss‚2.6605840128228624 2loss/SRqqt1/assignment_loss‚0.7390122447172274 1loss/SRqqt1/detection_loss‚1.2032971426151593 /loss/FBt1/detection_loss‚0.8952944340074763 &loss/total_loss‚11.960632627005413 trainer/global_step‚1949 % _timestamp‚1.7848427778293204e+09 1loss/SRqqt2/assignment_loss‚0.703775904708667 /loss/FRt1/detection_loss‚0.6230353456571035 /loss/FRt2/detection_loss‚1.5878740319322502 _step‚161¡ Ovž“è ‚(:ˆ §”ŠÓÖÝå³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚278.859' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.953 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xia×<é ‚(:±  §”ŠÓéæ‚´ disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0$ memory_percent‚88.97132873535156 disk.drbd1356.out‚0 network.sent‚ 3.460922e+06 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 network.recv‚251250) disk./.usagePercent‚70.51313783918327% disk./.usageGB‚1260.8465194702148" disk.dm-0.out‚11747.3330078125 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 97.31640625 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.740223566691082 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.02734375 disk.drbd1552.out‚0 disk.sda.out‚ 97.31640625 disk.drbd2111.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚11747.3330078125 disk.drbd2041.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚58.625# proc.memory.rssMB‚ 2885.27734375 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.rbd1.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd1938.out‚0 disk.dm-0.in‚58.625 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 cpu‚25.267679681004054 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.rbd1.out‚ 15.482421875 disk.drbd1584.in‚0& proc.memory.availableMB‚ 2710.40625è~L‰œê ‚(:‘ ¶”ŠÓÌéʳ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.174' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.72466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªòùJ:ë ‚(:¯  ¶”ŠÓóþë³ disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 15.544921875 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1356.in‚0 disk.sda2.in‚ 0.02734375 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 network.recv‚251358 disk.nvme0n1.in‚58.625 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚58.625 disk.drbd2078.out‚0 disk.rbd1.in‚0 cpu‚25.275378616946384 proc.cpu.threads‚17$ memory_percent‚88.97207578023276 network.sent‚ 3.46103e+06 disk.drbd1226.in‚0 disk.drbd1226.out‚0! disk.dm-0.out‚11748.458984375 disk.sda.out‚ 97.6328125 disk.drbd1938.out‚0$ disk.nvme0n1.out‚11748.458984375 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1285.in‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 2710.22265625* proc.memory.percent‚11.740223566691082) disk./.usagePercent‚70.51313997256167% disk./.usageGB‚1260.8465576171875 disk.sda2.out‚ 97.6328125 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.02734375«Åº2ì - _wandbruntime‚3578  _runtime‚3578œ/è¢b&í „x +particle/specificity‚0.9009441137313843 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868435859680176 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5730668306350708 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7684744000434875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7680541276931763 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5281059741973877 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6004538536071777 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19004079699516296 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14180776476860046 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24440792202949524 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9965009689331055 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7159775495529175 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7735235095024109 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5549644827842712 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1533428430557251 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16453926265239716 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2083333283662796 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.5814834237098694 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6292690634727478 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.560734212398529 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6227198839187622 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.691307008266449 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030559604987502098 _step‚162 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9967160820960999 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1649540364742279 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12508369982242584 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916073679924011 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.187398299574852 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6146978735923767 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9780768156051636 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906244874000549 particle/accuracy_1_of_4‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5879948735237122 particle/accuracy_2_of_4‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8357152938842773 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931122660636902 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7665739059448242 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5023373365402222 +jet/accuracy_2_of_3‚0.47092700004577637 *jet/accuracy_2_of_4‚0.7967772483825684 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23942647874355316 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5558263659477234 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17635641992092133 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9946590065956116 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6886830925941467 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.624235212802887 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6403132081031799 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6188584566116333 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13767556846141815 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08876003324985504 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.778703510761261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.04993041232228279 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3999583423137665 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1902938336133957 +jet/accuracy_4_of_4‚0.02385834977030754 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8958333134651184 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.41667914390563965 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 *jet/accuracy_1_of_1‚0.2848232686519623 6validation_average_jet_accuracy‚0.3313210904598236 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9886239767074585 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5097634792327881 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5269284844398499 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17438925802707672 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6247162818908691 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9922335743904114 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7600051164627075 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5547311902046204 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.71270751953125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4583333432674408 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7484849691390991 *jet/accuracy_1_of_4‚0.9687690734863281 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12639522552490234 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4970470368862152 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7501501441001892 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34482356905937195 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5982428789138794 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18563687801361084 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9814599752426147 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7834106087684631 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_2‚1 % _timestamp‚1.7848428149074457e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17737793922424316 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15117870271205902 /particle/accuracy_4_of_4‚0.9534932374954224 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5001445412635803 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015176168642938137 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928564429283142 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1700725257396698 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17733336985111237 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17315423488616943 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25709161162376404 'particle/accuracy‚0.832403302192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917625188827515 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2461538463830948 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 particle/accuracy_1_of_3‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6119806170463562 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4820513129234314 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49242421984672546 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982143044471741 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9086862206459045 /particle/accuracy_3_of_3‚0.9992997050285339 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42827102541923523 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9965009689331055 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49179503321647644 7 Purity/0FRt0SRqqt*FBt/FBt_pjq¦¨urity‚0.9930959939956665 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15623272955417633 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8391117453575134 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18858648836612701 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9766042828559875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003762003732845187 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7841890454292297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.12351690232753754 +jet/accuracy_3_of_4‚0.10053626447916031 particle/accuracy_2_of_2‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8699206113815308 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5664381384849548 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9874573945999146 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5508422255516052 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚1959 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11577226966619492 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5531089305877686 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9807513952255249 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2662584185600281 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9816418886184692 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12616309523582458 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0018526485655456781 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.7857142686843872 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08876003324985504 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6998290419578552 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3324715793132782 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932869672775269 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.560734212398529 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17733336985111237 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16457915306091309 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.546583354473114 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.53130704164505 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14616404473781586 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 epoch‚6 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2692928612232208 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9702522158622742 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03590226173400879 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45882681012153625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2612147033214569 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895530939102173 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3322559893131256 *loss/val_total_loss‚12.174493914600765 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8796935677528381 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4931093156337738 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2332129329442978 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13767030835151672 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11396050453186035 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851261377334595 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14418092370033264 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5344809889793396 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.778703510761261 'particle/f_score‚0.5981956124305725 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7679267525672913 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04173022508621216 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11611723154783249 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 _runtime‚ 3578.26549187ë€æ¿<î ‰x *jet/accuracy_1_of_1‚0.2848232686519623 6validation_average_jet_accuracy‚0.3313210904598236 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9886239767074585 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5097634792327881 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5269284844398499 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17438925802707672 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6247162818908691 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9922335743904114 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7600051164627075 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5547311902046204 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.71270751953125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4583333432674408 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7484849691390991 *jet/accuracy_1_of_4‚0.9687690734863281 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12639522552490234 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4970470368862152 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7501501441001892 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34482356905937195 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5982428789138794 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18563687801361084 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9814599752426147 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7834106087684631 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_2‚1 % _timestamp‚1.7848428149074457e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17737793922424316 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15117870271205902 /particle/accuracy_4_of_4‚0.9534932374954224 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5001445412635803 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015176168642938137 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928564429283142 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1700725257396698 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17733336985111237 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17315423488616943 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25709161162376404 'particle/accuracy‚0.832403302192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917625188827515 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2461538463830948 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 particle/accuracy_1_of_3‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6119806170463562 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4820513129234314 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49242421984672546 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982143044471741 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9086862206459045 /particle/accuracy_3_of_3‚0.9992997050285339 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42827102541923523 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9965009689331055 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49179503321647644 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930959939956665 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15623272955417633 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8391117453575134 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18858648836612701 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9766042828559875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003762003732845187 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7841890454292297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.12351690232753754 +jet/accuracy_3_of_4‚0.10053626447916031 particle/accuracy_2_of_2‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8699206113815308 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5664381384849548 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9874573945999146 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5508422255516052 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚1959 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11577226966619492 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5531089305877686 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9807513952255249 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2662584185600281 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9816418886184692 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12616309523582458 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0018526485655456781 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.7857142686843872 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08876003324985504 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6998290419578552 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3324715793132782 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932869672775269 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.560734212398529 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17733336985111237 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16457915306091309 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.546583354473114 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.53130704164505 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14616404473781586 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 epoch‚6 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2692928612232208 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9702522158622742 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03590226173400879 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45882681012153625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2612147033214569 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895530939102173 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3322559893131256 *loss/val_total_loss‚12.174493914600765 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8796935677528381 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4931093156337738 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2332129329442978 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13767030835151672 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11396050453186035 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851261377334595 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14418092370033264 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5344809889793396 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.778703510761261 'particle/f_score‚0.5981956124305725 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7679267525672913 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04173022508621216 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11611723154783249 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 _runtime‚ 3578.26549187 +particle/specificity‚0.9009441137313843 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868435859680176 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5730668306350708 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7684744000434875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7680541276931763 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5281059741973877 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6004538536071777 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19004079699516296 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14180776476860046 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24440792202949524 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9965009689331055 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7159775495529175 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7735235095024109 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5549644827842712 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1533428430557251 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16453926265239716 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2083333283662796 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.5814834237098694 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6292690634727478 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.560734212398529 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6227198839187622 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.691307008266449 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030559604987502098 _step‚162 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9967160820960999 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1649540364742279 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12508369982242584 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916073679924011 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.187398299574852 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6146978735923767 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9780768156051636 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906244874000549 particle/accuracy_1_of_4‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5879948735237122 particle/accuracy_2_of_4‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8357152938842773 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931122660636902 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7665739059448242 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5023373365402222 +jet/accuracy_2_of_3‚0.47092700004577637 *jet/accuracy_2_of_4‚0.7967772483825684 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23942647874355316 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5558263659477234 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17635641992092133 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9946590065956116 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6886830925941467 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.624235212802887 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6403132081031799 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6188584566116333 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13767556846141815 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08876003324985504 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.778703510761261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.04993041232228279 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3999583423137665 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1902938336133957 +jet/accuracy_4_of_4‚0.02385834977030754 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8958333134651184 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.41667914390563965 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096¢Õ"Ÿžï ‚(:“ Å”ŠÓŸäÓ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚262.324' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.44133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ”Jk5ð ‚(:ª  Å”ŠÓŸ•ó³% disk./.usageGB‚1260.8503189086914 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2885.27734375 network.recv‚252320 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51335032367277 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚ 0.02734375! disk.dm-0.out‚12718.212890625$ disk.nvme0n1.out‚12718.212890625 disk.drbd1931.in‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 2913.27734375 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚58.625 network.sent‚ 3.466568e+06 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0$ memory_percent‚88.14584414164224 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚98.1875 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.740223566691082 disk.drbd2041.out‚0 disk.rbd1.out‚ 15.544921875 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚58.625 disk.drbd2118.in‚0 disk.sda.in‚ 0.02734375 disk.drbd1356.in‚0 cpu‚25.277414291713317 disk.sda.out‚98.1875 disk.drbd1289.out‚0 disk.drbd2111.out‚0ÝÍ`<ñ ‚(:±  Ô”ŠÓ’ÀŠ´ disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚12718.87890625 cpu‚25.277436058953512% disk./.usageGB‚1260.8503189086914 disk.sda.out‚ 98.91796875 disk.dm-0.out‚12718.87890625 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚ 0.02734375 disk.dm-0.in‚58.625 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 15.5771484375 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚58.625* proc.memory.percent‚11.740223566691082 disk.sda1.out‚0 network.sent‚ 3.531893e+06 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 98.91796875 disk.drbd2041.out‚0) disk./.usagePercent‚70.51335032367277 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 2913.52734375 proc.cpu.threads‚17$ memory_percent‚88.14482688903809 network.recv‚254795 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.sda.in‚ 0.02734375 disk.drbd1673.out‚0 disk.drbd1289.in‚0ÛŽóò ‚(:’ Ô”ŠÓ¸³ç³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚148.85' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚49.61666666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žò}Ÿó ‚(:” 㔊Óüüϳ gpu.0.gpu‚55.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚302.845' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.94833333333335 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é£=m:ô ‚(:¯  㔊Ó¾¿î³ disk.sda.in‚ 0.02734375 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.740223566691082 disk.drbd1931.in‚0 disk.sda.out‚ 99.3046875 disk.sda2.in‚ 0.02734375 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51313783918327 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 2913.27734375 disk.dm-0.in‚58.625 disk.drbd2078.in‚0 disk.drbd2041.in‚0# disk.nvme0n1.out‚13688.76171875 disk.sda1.in‚0 cpu‚25.277531996961002 network.recv‚254861 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1550.in‚0 network.sent‚ 3.531959e+06% disk./.usageGB‚1260.8465194702148 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0$ memory_percent‚88.14584414164224 disk.drbd1817.out‚0 disk.sda2.out‚ 99.3046875 disk.drbd1226.in‚0 disk.nvme0n1.in‚58.625 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.dm-0.out‚13688.76171875 disk.drbd1965.in‚0 disk.rbd1.out‚ 15.6708984375 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0•-uÑžõ ‚(:“ ò”ŠÓþ÷ѳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚275.656' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.88533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†€I?ö ‚(:´  ò”ŠÓÝñ³ disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 2913.46484375 disk.drbd1584.in‚0 disk.sda.in‚ 0.02734375 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 15.7333984375 disk.sda.out‚ 99.5859375 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0$ memory_percent‚88.14508120218912 disk.sda2.in‚ 0.02734375 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8465385437012 disk.drbd1817.out‚0 disk.sda2.out‚ 99.5859375 disk.dm-0.in‚58.625 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51313890587248 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1550.in‚0 cpu‚25.277350737222495 network.recv‚263383 disk.drbd1817.in‚0 disk.nvme0n1.in‚58.625 disk.drbd1673.in‚0* proc.memory.percent‚11.740223566691082 network.sent‚ 3.59013e+06 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0# disk.dm-0.out‚13689.11962890625# proc.memory.rssMB‚ 2885.27734375 disk.drbd2041.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚13689.11962890625 disk.drbd1510.out‚0 disk.drbd1550.out‚0rY˜’÷ ‚(:‡ •ŠÓ½íÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.64' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.88 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o<ø ‚(:±  •ŠÓÉéý³ proc.cpu.threads‚17# disk.dm-0.out‚13689.35009765625 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125' proc.memory.availableMB‚ 2913.203125 disk.nvme0n1.in‚58.625 disk.drbd1584.in‚0 disk.drbd1931.in‚0 cpu‚25.277305743173784) disk./.usagePercent‚70.51313911921032 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.rbd1.out‚ 15.7333984375 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 100.82421875 network.sent‚ 3.590196e+06 network.recv‚263449 disk.drbd1828.out‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 100.82421875 disk.drbd2041.out‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚13689.35009765625 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚58.625 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0% disk./.usageGB‚1260.8465423583984 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0$ memory_percent‚88.14614613850912* proc.memory.percent‚11.740223566691082 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125ü¥·‘2ù - _wandbruntime‚3651  _runtime‚3651n´0AÇú Á _step‚163 ' lr-AdamW/pg1‚0.00011265051514126154 ' lr-AdamW/pg2‚0.00011265051514126154 trainer/global_step‚1999 $ _timestamp‚1.784842888164077e+09 _runtime‚ 3651.52130583>ôÆÌû Æ trainer/global_step‚1999 $ _timestamp‚1.784842888164077e+09 _runtime‚ 3651.52130583 _step‚163 ' lr-AdamW/pg1‚0.00011265051514126154 ' lr-AdamW/pg2‚0.00011265051514126154£`<N2ü - _wandbruntime‚3651  _runtime‚3651]†«ý ìO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5332947 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2084205346888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2082860735602 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18565897 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11427449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 E8DeviceStatsMonißÛf‡G#tor.on_train_batch_start/active.all.freed‚35930329 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚577917731100672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5332952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577917731100672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2086355070976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚575831376029696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11427113 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6094166 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _step‚164 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18565897 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18575961 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157706923622400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160259312264704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2552439909376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17364438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575603323332272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17364438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17364432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2087700507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2086355070976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2552388642304 $ _timestamp‚1.784842888165145e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160259731622912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚577919102710272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚1999 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚575831402203136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚575831376029696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575603348760240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6094497 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577687554107128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚575831402203136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17364432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35940399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18575961 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577686184067874 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0›]u¬÷'þ ñO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚575831402203136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17364432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35940399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18575961 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577686184067874 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5332947 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2084205346888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2082860735602 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18565897 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11427449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35930329 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚577917731100672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5332952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577917731100672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2086355070976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚575831376029696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11427113 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6094166 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _step‚164 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18565897 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18575961 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157706923622400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160259312264704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2552439909376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17364438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575603323332272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17364438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17364432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2087700507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2086355070976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2552388642304 $ _timestamp‚1.784842888165145e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160259731622912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚577919102710272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚1999 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚575831402203136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚575831376029696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575603348760240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6094497 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577687554107128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755¤dW%22ÿ - _wandbruntime‚3652  _runtime‚3652;œ˜½ý&€ ÷M ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576110311493632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3652.995210831 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576110311493632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5335592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575882150534016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35947778 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18574979 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17372805 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17372799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578197662899200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35957862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18585057 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35947778 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577966006643290 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35957862 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2088723022336 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2085226148640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576110285320192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 trainer/global_step‚1999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577967376682656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157785509988352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2553635925504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11432669 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6097413 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2087377579008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17372799 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157785878079488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2553687185408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576110285320192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2088723022336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6097082 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578199034515968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚165 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18585057 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2083881537242D È(p  üM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576110311493632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3652.995210831 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576110311493632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5335592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575882150534016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35947778 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18574979 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17372805 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17372799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578197662899200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35957862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18585057 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35947778 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577966006643290 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35957862 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_tr\¤¿¬’ain_batch_end/active_bytes.small_pool.allocated‚ 2088723022336 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2085226148640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576110285320192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 trainer/global_step‚1999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577967376682656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157785509988352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2553635925504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11432669 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6097413 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2087377579008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17372799 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157785878079488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2553687185408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576110285320192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2088723022336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6097082 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578199034515968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚165 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18585057 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2083881537242¥Ó7Á“2‚ - _wandbruntime‚3653  _runtime‚3653±ö:ƒ þ 1loss/FBt1/assignment_loss‚0.23354863424538053 /loss/FRt1/detection_loss‚0.6597292406629042 /loss/FBt1/detection_loss‚2.2735979203134775 epoch‚7 % _timestamp‚1.7848428896752772e+09 /loss/FRt2/assignment_loss‚1.247356447714642 1loss/FBt2/assignment_loss‚0.07987383457055021 .loss/FRt2/detection_loss‚1.532704661758073 1loss/SRqqt2/detection_loss‚0.7010326916841042 %loss/total_loss‚12.82127397663995 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚0.7946219185899411 1loss/SRqqt1/detection_loss‚1.2483523538469203 /loss/FBt2/detection_loss‚0.7422572866258996 _step‚166 2loss/SRqqt2/assignment_loss‚0.6487477849467966 _runtime‚3653.032493499 /loss/FRt1/assignment_loss‚2.659451201681263BX¢· „ ƒ /loss/FBt1/detection_loss‚2.2735979203134775 epoch‚7 % _timestamp‚1.7848428896752772e+09 /loss/FRt2/assignment_loss‚1.247356447714642 1loss/FBt2/assignment_loss‚0.07987383457055021 .loss/FRt2/detection_loss‚1.532704661758073 1loss/SRqqt2/detection_loss‚0.7010326916841042 %loss/total_loss‚12.82127397663995 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚0.7946219185899411 1loss/SRqqt1/detection_loss‚1.2483523538469203 /loss/FBt2/detection_loss‚0.7422572866258996 _step‚166 2loss/SRqqt2/assignment_loss‚0.6487477849467966 _runtime‚3653.032493499 /loss/FRt1/assignment_loss‚2.659451201681263 1loss/FBt1/assignment_loss‚0.23354863424538053 /loss/FRt1/detection_loss‚0.6597292406629042¦í5;²ž… ‚(:“ •ŠÓ™ªÛ³ gpu.0.gpu‚59.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.798' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.93266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ±Þ“+† ‚(:   •ŠÓàíû³ disk.drbd2111.in‚0 disk.sda2.out‚ 101.2578125 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0! memory_percent‚88.14697265625% disk./.usageGB‚1260.8465423583984 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 proc.memory.availableMB‚2913) disk./.usagePercent‚70.51313911921032 disk.drbd1828.in‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.sda1.out‚0* proc.memory.percent‚11.740223566691082# disk.dm-0.out‚13689.80322265625 disk.sda.out‚ 101.2578125 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.dm-0.in‚58.625 disk.drbd1965.in‚0 disk.nvme0n1.in‚58.625 disk.drbd2111.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚13689.80322265625 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 15.765625 network.sent‚ 3.595893e+06 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 cpu‚25.27733044165498 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1289.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 network.recv‚264560 disk.drbd1285.in‚0V°e ‚(:ˆ Ÿ•ŠÓÙã³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚266.766' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£-Ù7ˆ ‚(:¬  Ÿ•ŠÓñà€´ proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.sda.out‚ 101.61328125 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.740223566691082) disk./.usagePercent‚70.51313805252111# disk.dm-0.out‚13689.98681640625 disk.drbd1226.out‚0) proc.memory.availableMB‚ 2912.98828125 disk.sda2.out‚ 101.61328125 disk.drbd1965.out‚0 disk.rbd1.out‚16.03125 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2885.27734375$ disk./.usageGB‚1260.846523284912 disk.sda2.in‚0.03125 disk.dm-0.in‚58.625 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 network.recv‚264668 disk.drbd1965.in‚0 disk.nvme0n1.in‚58.625 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1817.in‚0 network.sent‚ 3.596001e+06 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚13689.98681640625 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 cpu‚25.27735422238558 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.sda1.out‚0$ memory_percent‚88.14702033996582ey™’ž‰ ‚(:“ ®•ŠÓаϳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚157.006' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚52.33533333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rÜ!g6Š ‚(:«  ®•ŠÓÚŒî³ disk.rbd1.out‚16.09375* proc.memory.percent‚11.740223566691082 network.sent‚ 3.596109e+06 disk.dm-0.in‚58.625 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚13690.31982421875 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 102.03515625 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚58.625 disk.drbd1552.in‚0 disk.sda.out‚ 102.03515625 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2885.27734375$ disk./.usageGB‚1260.846523284912 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125' proc.memory.availableMB‚ 2912.984375 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125) disk./.usagePercent‚70.51313805252111 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 network.recv‚264776 disk.drbd1817.in‚0$ memory_percent‚88.14703623453775 cpu‚25.277313252850025 disk.drbd1289.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0# disk.dm-0.out‚13690.31982421875 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0ƒyDõž‹ ‚(:“ ½•ŠÓÎïͳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.196' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.06533333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ék=ƒ3Œ ‚(:¨  ½•ŠÓŒ„ì³ disk.drbd1931.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 network.recv‚265774 disk.drbd2118.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚16.09375 disk.drbd1285.out‚0 disk.drbd1510.out‚0# disk.dm-0.out‚13690.53466796875( disk./.usagePercent‚70.5131124519802 disk.sda.out‚ 102.453125 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.740223566691082 disk.nvme0n1.in‚58.625 disk.sda2.out‚ 102.453125 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1289.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.sda.in‚0.03125 network.sent‚ 3.601694e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1356.in‚0$ memory_percent‚88.14713160196939 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1552.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚13690.53466796875 disk.drbd1550.out‚0( proc.memory.availableMB‚ 2912.9609375 cpu‚25.277270698483484 disk.drbd1938.out‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8460655212402 disk.drbd1931.in‚0 disk.dm-0.in‚58.625 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1226.out‚0¥q¦ÊŸ ‚(:” Ì•ŠÓ¦«è³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.063' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.68766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œð!·:Ž ‚(:¯  Ì•ŠÓ»Ãˆ´( disk./.usagePercent‚70.5131124519802 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0$ memory_percent‚88.14713160196939 disk.drbd2041.out‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8460655212402 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0( proc.memory.availableMB‚ 2912.9609375* proc.memory.percent‚11.740223566691082 disk.drbd1285.out‚0& disk.nvme0n1.out‚13690.83935546875 disk.drbd1550.out‚0 cpu‚25.277291815994197 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1289.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 102.8203125 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 16.1259765625 disk.drbd1931.out‚0 disk.nvme0n1.in‚58.625 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.dm-0.in‚58.625 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚265924 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 network.sent‚ 3.601844e+06 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0# disk.dm-0.out‚13690.83935546875 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1552.in‚0 disk.sda.out‚ 102.8203125 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0# proc.memory.rssMB‚ 2885.27734375v)ò” ‚(:‰ Û•ŠÓù¤â³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.455' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.485 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µî¶v8 ‚(:­  Û•ŠÓóèö³ disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚13691.02294921875 disk.drbd1550.in‚0 disk.drbd2118.in‚0# disk.dm-0.out‚13691.02294921875( proc.memory.availableMB‚ 2913.2109375# proc.memory.rssMB‚ 2885.27734375 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 network.sent‚ 3.666972e+06% disk./.usageGB‚1260.8460273742676 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 103.203125$ memory_percent‚88.14611434936523* proc.memory.percent‚11.740223566691082 disk.drbd1817.out‚0 disk.nvme0n1.in‚58.625 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 cpu‚25.277516777105458 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 103.203125 disk.drbd1931.in‚0 network.recv‚267359 disk.drbd1673.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.dm-0.in‚58.625 disk.drbd1285.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1965.in‚0( disk./.usagePercent‚70.5131103186018 disk.drbd2111.out‚0 disk.rbd1.out‚ 16.2197265625 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0-ê¾2‘ - _wandbruntime‚3737  _runtime‚3737} þÉ’ à ' lr-AdamW/pg1‚0.00011238981863961578 ' lr-AdamW/pg2‚0.00011238981863961578 trainer/global_step‚2049 % _timestamp‚1.7848429746266937e+09 _runtime‚3737.983947432 _step‚167ÝsŸÎ“ È ' lr-AdamW/pg2‚0.00011238981863961578 trainer/global_step‚2049 % _timestamp‚1.7848429746266937e+09 _runtime‚3737.983947432 _step‚167 ' lr-AdamW/pg1‚0.00011238981863961578§±†Aa2” - _wandbruntime‚3737  _runtime‚3737s剂ð'• êO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591915756791296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164251253683712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19031913 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚589543412021072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6241274 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591914385181696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589776840554496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11706221 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5464947 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164251673041920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2135309229696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5464952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17782788 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚591915756791296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2138890063360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2137544627200 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19021849 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2138890063360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17782788 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17782782 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚359 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2615063030784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2133964618410 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6241633 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19021849 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591914385181696 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589776866727936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161636610011136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161636241920000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2615011763712 " _timestamp‚1.7848429746278e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36814701 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3737.985304668 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050-›ÓÍ– ïO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19031913 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚589543412021072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6241274 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591914385181696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589776840554496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11706221 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5464947 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164251673041920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2135309229696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.¸ç(!on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5464952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17782788 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚591915756791296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2138890063360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2137544627200 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19021849 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2138890063360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17782788 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17782782 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚359 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2615063030784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2133964618410 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6241633 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19021849 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591914385181696 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589776866727936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161636610011136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161636241920000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2615011763712 " _timestamp‚1.7848429746278e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36814701 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3737.985304668 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591915756791296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164251253683712¨¥—˜Õ2— - _wandbruntime‚3739  _runtime‚37396Š¡aü&˜ öM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2134985420050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17791149 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36822080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592195688596992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2138567135232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5467587 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17791155 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19041009 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19041009 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11711810 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591958569254264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2136330031448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _step‚169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2138567135232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19030931 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2616320759808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3739.460993822 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17791155 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36822080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6244223 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161714828285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616267402752 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36832164 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2139912578560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590055749844992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚589822239222816 trainer/global_step‚2049 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590055776018432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17791149 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6244572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2139912578560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161715196377088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591957199214898 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19030931 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17791149 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36822080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592195688596992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2138567135232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5467587 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17791155 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19041009 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19041009 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11711810 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591958569254264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2136330031448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _step‚169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2138567135232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19030931 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2616320759808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3739.460993822 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17791155 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36822080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6244223 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161714828285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616267402752 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36832164 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2139912578560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590055749844992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚589822239222816 trainer/global_step‚2049 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590055776018432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17791149 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6244572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2139912578560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161715196377088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591957199214898 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19030931 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E^ŸŸ ‚(:” ù•ŠÓ¼•¸³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.826' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.27533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çé"°  ‚(:­  ù•ŠÓÜØÔ³) disk./.usagePercent‚70.51321570749518 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sdaºËÏ8ˆ1.out‚0 network.sent‚ 3.672598e+06 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚58.625 disk.drbd2041.in‚0$ disk.nvme0n1.out‚13691.955078125 disk.drbd1552.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.740223566691082# proc.memory.rssMB‚ 2885.27734375 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 cpu‚25.277369030152638 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1285.in‚0) proc.memory.availableMB‚ 2912.70703125 disk.drbd1289.in‚0! disk.dm-0.out‚13691.955078125 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 103.7578125 disk.drbd1965.out‚0 disk.sda1.in‚0 network.recv‚268366% disk./.usageGB‚1260.8479118347168 disk.drbd1584.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 103.7578125 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0$ memory_percent‚88.14816474914551 disk.rbd1.out‚ 16.2822265625 disk.drbd1285.out‚0 disk.nvme0n1.in‚58.625 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0ׄ¤¤Ÿ¡ ‚(:” ˆ–ŠÓð˳³ gpu.0.gpu‚54.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.982' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.32733333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M¸3¢ ‚(:¨  ˆ–ŠÓªÿг disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.sda.in‚0.03125% disk./.usageGB‚1260.8478622436523 disk.drbd1817.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 16.314453125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0$ memory_percent‚88.14829190572104 disk.drbd1552.in‚0 disk.drbd1931.out‚0 network.recv‚268474) disk./.usagePercent‚70.51321293410325 disk.sda1.in‚0 disk.drbd2118.in‚0& disk.nvme0n1.out‚13692.46533203125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚58.625* proc.memory.percent‚11.740223566691082 disk.sda.out‚104.125 disk.drbd1673.in‚0# disk.dm-0.out‚13692.46533203125 disk.drbd1356.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 2912.67578125 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚104.125 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1356.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚58.625 disk.drbd1931.in‚0 disk.sda2.in‚0.03125 network.sent‚ 3.672706e+06 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1673.out‚0 disk.drbd1552.out‚0 cpu‚25.277327341149793 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd2111.in‚0ã)iž£ ‚(:“ —–ŠÓ¿÷¨³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.945' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.98166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íåáÓ;¤ ‚(:°  —–ŠÓþŒË³) disk./.usagePercent‚70.51319970715711 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2885.27734375 network.sent‚ 3.735367e+06 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.dm-0.in‚58.625 disk.drbd1817.in‚0 disk.drbd1673.in‚0% disk./.usageGB‚1260.8476257324219 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 cpu‚25.277286218111353 disk.drbd2111.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0& disk.nvme0n1.out‚13692.95361328125 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 104.4296875 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 2912.67578125 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 104.4296875# disk.dm-0.out‚13692.95361328125 disk.drbd1965.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0$ memory_percent‚88.14829190572104 network.recv‚270041 disk.drbd1965.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.740223566691082 proc.cpu.threads‚17 disk.rbd1.out‚ 16.408203125 disk.nvme0n1.in‚58.625 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0JÞUbž¥ ‚(:“ ¦–ŠÓŒçã³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.402' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.80066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I=º:¦ ‚(:¯  ¦–ŠÓý¹ƒ´ disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚13693.42724609375 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1285.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.nvme0n1.in‚58.625 disk.sda1.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 104.7109375 disk.sda2.out‚ 104.7109375 disk.drbd1356.out‚0 disk.rbd1.in‚0 cpu‚25.27723868028247 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.740223566691082 network.recv‚270107% disk./.usageGB‚1260.8476753234863 disk.drbd1550.in‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2885.27734375) disk./.usagePercent‚70.51320248054905 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 2912.67578125 disk.dm-0.in‚58.625 disk.drbd1356.in‚0 disk.rbd1.out‚ 16.470703125 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 network.sent‚ 3.735433e+06 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0$ memory_percent‚88.14829190572104 disk.drbd2118.out‚0& disk.nvme0n1.out‚13693.42724609375 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0ÑbÎ>2§ - _wandbruntime‚3824  _runtime‚3824í[”AȨ  trainer/global_step‚2099 $ _timestamp‚1.784843061424442e+09 _runtime‚3824.781685354 _step‚171 ' lr-AdamW/pg1‚0.00011212206687340549 ' lr-AdamW/pg2‚0.00011212206687340549"Ÿç²Í© Ç ' lr-AdamW/pg1‚0.00011212206687340549 ' lr-AdamW/pg2‚0.00011212206687340549 trainer/global_step‚2099 $ _timestamp‚1.784843061424442e+09 _runtime‚3824.781685354 _step‚171«S–‚@2ª - _wandbruntime‚3824  _runtime‚3824ÿøoRò'« ìO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168243545076736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603483526137840 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603483500709872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605911039262720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5596947 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2190079619584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37689003 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603722331252736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2188734183424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19477801 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5596952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18201138 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168243123621376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚2099÷!ßú÷'¬ ñO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605911039262720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5596947 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2190079619584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37689003 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603722331252736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2188734183424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19477801 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5596952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18201138 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168243123621376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚2099 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168243545076736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603483526137840 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603483500709872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11990163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2187433914256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37706466 FDeviceStatsMonitor.on_train_batch_end/allocateÃu²_d_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2191102134784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚3826.263498949 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2186089302858 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6390941 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19496961 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165644514674688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37696382 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168323384764416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37706466 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19486883 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605948391786506 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18209505 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605949761825872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11990163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2187433914256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37706466 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2191102134784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚3826.263498949 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2186089302858 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6390941 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19496961 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165644514674688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37696382 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168323384764416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37706466 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19486883 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605948391786506 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005­0]2² - _wandbruntime‚3826  _runtime‚3826m½£³ þ 0loss/FRt2/assignment_loss‚1.0052101781016671 2loss/SRqqt1/assignment_loss‚0.6912223980379021 %loss/total_loss‚11.98458394572684 % _timestamp‚1.7848430629390874e+09 1loss/SRqqt2/assignment_loss‚0.660316937188161 1loss/FBt1/assignment_loss‚0.06345267394256855 /loss/FRt2/detection_loss‚1.6434846861333015 1loss/SRqqt2/detection_loss‚0.7824893751137882 0loss/FRt1/assignment_loss‚2.6146876448451883 /loss/FRt1/detection_loss‚0.6039868475166196 epoch‚7 _runtime‚3826.296311362 _step‚174 0loss/FBt2/assignment_loss‚0.1823778601295799 0loss/SRqqt1/detection_loss‚1.080492300892459 /loss/FBt1/detection_loss‚1.8951310930618395 /loss/FBt2/detection_loss‚0.7617319507637645 trainer/global_step‚2099¡Œwp ´ ƒ 1loss/SRqqt2/assignment_loss‚0.660316937188161 1loss/FBt1/assignment_loss‚0.06345267394256855 /loss/FRt2/detection_loss‚1.6434846861333015 1loss/SRqqt2/detection_loss‚0.7824893751137882 0loss/FRt1/assignment_loss‚2.6146876448451883 /loss/FRt1/detection_loss‚0.6039868475166196 epoch‚7 _runtime‚3826.296311362 _step‚174 0loss/FBt2/assignment_loss‚0.1823778601295799 0loss/SRqqt1/detection_loss‚1.080492300892459 /loss/FBt1/detection_loss‚1.8951310930618395 /loss/FBt2/detection_loss‚0.7617319507637645 trainer/global_step‚2099 0loss/FRt2/assignment_loss‚1.0052101781016671 2loss/SRqqt1/assignment_loss‚0.6912223980379021 %loss/total_loss‚11.98458394572684 % _timestamp‚1.7848430629390874e+09®ÕãÚš9µ ‚(:®  Ä–ŠÓäëð³$ memory_percent‚88.14849853515625) disk./.usagePercent‚70.51320354723826 disk.drbd1510.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.rbd1.out‚ 16.5029296875 disk.drbd1673.out‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.740223566691082 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚13694.37939453125 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 105.46484375 disk.dm-0.in‚58.625 disk.drbd1550.in‚0$ proc.memory.availableMB‚2912.625 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 network.recv‚271199 disk.drbd1828.out‚0 disk.sda.out‚ 105.46484375 disk.rbd1.in‚0 disk.nvme0n1.in‚58.625 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚13694.37939453125 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 cpu‚25.277158792816113 disk.sda1.out‚0 disk.drbd2041.out‚0 network.sent‚ 3.741057e+06% disk./.usageGB‚1260.8476943969727 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0i—;‡ž¶ ‚(:“ Ä–ŠÓÍ£Õ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.717' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.57233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HD¦“· ‚(:ˆ Ó–ŠÓàùγ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.583' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.861 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜyÂ`8¸ ‚(:­  Ó–ŠÓžýì³ disk.drbd2118.out‚0 cpu‚25.277118736873664 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.nvme0n1.in‚58.625 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.sda2.out‚ 106.04296875 disk.drbd1285.in‚0 disk.rbd1.out‚ 16.5966796875 network.recv‚271307 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 106.04296875 disk.drbd1931.in‚0 network.sent‚ 3.741165e+06 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0$ proc.memory.availableMB‚2912.875( disk./.usagePercent‚70.5132163475087% disk./.usageGB‚1260.8479232788086 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125& disk.nvme0n1.out‚13694.63330078125 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.dm-0.in‚58.625# disk.dm-0.out‚13694.63330078125 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.740223566691082 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0$ memory_percent‚88.14748128255209 disk.sda1.in‚0 disk.drbd1938.out‚0ì~^õŸ¹ ‚(:” â–ŠÓ…‡ß³ gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.034' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.01133333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„§¾Z5º ‚(:ª  â–ŠÓ³Öù³ disk.drbd1931.in‚0! disk.dm-0.out‚13695.611328125 disk.drbd2078.out‚0$ disk.nvme0n1.out‚13695.611328125 disk.drbd1673.in‚0 disk.drbd1356.out‚0 cpu‚25.277077238167315% disk./.usageGB‚1260.8523559570312 disk.drbd1226.out‚0 disk.drbd1550.in‚0' proc.memory.availableMB‚ 3852.328125 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51346424607978 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.sda.out‚ 106.3359375 disk.sda.in‚0.03125 disk.nvme0n1.in‚58.625 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.rbd1.out‚ 16.6591796875 disk.drbd2118.in‚0 disk.sda2.out‚ 106.3359375 disk.dm-0.in‚58.625 disk.drbd1828.in‚0* proc.memory.percent‚11.740223566691082 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 network.sent‚ 3.74669e+06 network.recv‚272277 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0$ memory_percent‚84.32483673095703 disk.sda1.out‚0 disk.drbd1584.out‚0²ãd{ž» ‚(:“ ñ–ŠÓ˳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚267.247' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.08233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03´ócB¼ ‚(:·  ñ–ŠÓæ²ó³ disk.drbd1285.out‚0 disk.drbd1356.out‚0 disk.sda.out‚ 106.81640625 disk.drbd1673.out‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.rbd1.out‚ 16.6591796875 disk.drbd1552.in‚0 disk.drbd1552.out‚0$ memory_percent‚85.38557688395181 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0$ disk.nvme0n1.out‚13696.095703125 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.dm-0.in‚ 59.69140625 cpu‚25.277229520491602 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 106.81640625 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2885.27734375 network.sent‚ 3.746798e+06 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 network.recv‚272385% disk./.usageGB‚1260.8522605895996 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125! disk.dm-0.out‚13696.095703125 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0' proc.memory.availableMB‚ 3591.640625 disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 59.69140625 disk.drbd1550.in‚0* proc.memory.percent‚11.740223566691082) disk./.usagePercent‚70.51345891263375û³WÈž½ ‚(:“ €—ŠÓ¨´â³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚253.798' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.59933333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l8tB¾ ‚(:·  €—ŠÓĉ‰´ disk.drbd2111.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 3316.62890625 network.recv‚272451 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 cpu‚25.277186946190337 disk.drbd1584.in‚0) disk./.usagePercent‚70.51345165914717 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.sda2.in‚0.03125% disk./.usageGB‚1260.8521308898926 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 16.69140625 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚13697.462890625 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 59.69140625! disk.dm-0.out‚13697.462890625* proc.memory.percent‚11.740223566691082$ memory_percent‚86.50460243225098 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2885.27734375 disk.drbd1584.out‚0 network.sent‚ 3.746864e+06 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 107.23828125 disk.nvme0n1.in‚ 59.69140625 disk.sda2.out‚ 107.23828125õ(¯2¿ - _wandbruntime‚3911  _runtime‚3911iö~ÉÉÀ à % _timestamp‚1.7848431480890932e+09 _runtime‚3911.446334719 _step‚175 ' lr-AdamW/pg1‚0.00011184729493875274 ' lr-AdamW/pg2‚0.00011184729493875274 trainer/global_step‚2149ñ:ˆ^ÎÁ È trainer/global_step‚2149 % _timestamp‚1.7848431480890932e+09 _runtime‚3911.446334719 _step‚175 ' lr-AdamW/pg1‚0.00011184729493875274 ' lr-AdamW/pg2‚0.00011184729493875274¯†Å>2 - _wandbruntime‚3911  _runtime‚3911´ÄÍ÷£ à íO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169495246606336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38553235 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5728947 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619907693343744 WEDeviceStatsMonitor.on_train_batch_start/actEÆÑPive_bytes.small_pool.freed‚ 2239923739648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619909064953344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚3911.447714578 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617667769604096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2236172384026 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2739829383680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12263110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2739771825152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18619482 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5728952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172234650340352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19943817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚617423614826640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2237516995312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19933753 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6534531 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19943817 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2241269175808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18619488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2239923739648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12263483 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619907693343744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38563305 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169494878515200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619909064953344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617667795777536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚619659761782698 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚617667769604096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚617667795777536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19933753k )Ìø'Ä òO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2739771825152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18619482 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5728952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172234650340352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19943817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚617423614826640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2237516995312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19933753 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6534531 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19943817 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2241269175808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18619488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2239923739648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12263483 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619907693343744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38563305 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169494878515200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619909064953344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617667795777536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚619659761782698 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚617667769604096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚617667795777536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19933753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169495246606336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38553235 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5728947 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619907693343744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2239923739648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619909064953344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚3911.447714578 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617667769604096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2236172384026 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2739829383680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12263110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368° ô52Å - _wandbruntime‚3912  _runtime‚3912?øïRý&Æ ÷M XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2242291691008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2238537797064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6537072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19952913 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚177 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2741071926784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚617946678894592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18627855 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620188996759040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38570684 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2240946247680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19952913 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619939584358114 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731587 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6537429 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172314481353728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617946678894592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2240946247680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38570684 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18627849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚620187625142272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18627849 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12268659 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚619940954397480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚617702416600416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38580768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19942835 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚617702391172448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2237193185666 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2741016472576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18627855 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12269021 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169573464881152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617946705068032 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2240946247680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19952913 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619939584358114 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731587 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6537429 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172314481353728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617946678894592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2240946247680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38570684 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18627849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚620187625142272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18627849 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12268659 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚619940954397480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚617702416600416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38580768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19942835 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KBDevÓÀAœiceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚617702391172448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2237193185666 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2741016472576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18627855 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12269021 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169573464881152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617946705068032 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E;Ì ‚(:°  —ŠÓ˜ÎŒ¶ disk.drbd1226.in‚0* proc.memory.percent‚11.740223566691082 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 16.78515625 disk.drbd1285.out‚0 cpu‚25.27718303047713 disk.drbd1673.in‚0! disk.dm-0.out‚13697.638671875 network.sent‚ 3.812107e+06) disk./.usagePercent‚70.51345165914717 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.sda.out‚ 107.71875 disk.drbd2078.in‚0 disk.drbd1938.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 3039.90234375 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 107.71875 disk.dm-0.in‚ 59.69140625 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0$ memory_percent‚87.63060569763184 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 network.recv‚274307 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 59.69140625$ disk.nvme0n1.out‚13697.638671875 disk.drbd1550.out‚0% disk./.usageGB‚1260.8521308898926# proc.memory.rssMB‚ 2885.27734375 disk.drbd1673.out‚0 disk.drbd1828.in‚0¸fƒ©Í ‚(:’ ž—ŠÓ§òà³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚271.33' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.44333333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü<á#DÎ ‚(:¹  ž—ŠÓÚ·ƒ´ disk.drbd1673.in‚0 disk.sda.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 59.69140625 disk.sda1.in‚0 network.sent‚ 3.820319e+06 disk.drbd1938.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0( proc.memory.availableMB‚ 2749.6484375 disk.sda2.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 107.97265625 disk.sda2.out‚ 107.97265625 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 network.recv‚275495 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8521728515625 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0$ memory_percent‚88.81165186564127# disk.dm-0.out‚13698.53955078125 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 16.84765625& disk.nvme0n1.out‚13698.53955078125 disk.drbd1289.out‚0 cpu‚25.27704484012538 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 59.69140625* proc.memory.percent‚11.740223566691082) disk./.usagePercent‚70.51345400586341# proc.memory.rssMB‚ 2885.27734375 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2118.out‚0ú›+žÏ ‚(:“ ­—ŠÓÐ÷ª³ gpu.0.gpu‚20.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚321.179' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚107.05966666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥:¿\DÐ ‚(:¹  ­—ŠÓüÐγ network.recv‚275561 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51345400586341 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1226.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8521728515625 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.rbd1.out‚ 16.84765625 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 108.3203125 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 cpu‚25.277074001837825 disk.drbd1510.in‚0) proc.memory.availableMB‚ 2460.99609375* proc.memory.percent‚11.740223566691082 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 network.sent‚ 3.820385e+06# disk.dm-0.out‚13698.68798828125 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.dm-0.in‚ 59.69140625 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 59.69140625& disk.nvme0n1.out‚13698.68798828125 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 108.3203125 disk.drbd1931.in‚0$ memory_percent‚89.98618125915527¾áÊ3AÑ ‚(:¶  ¼—ŠÓ¸óѳ disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 108.63671875 disk.drbd1289.out‚0 disk.rbd1.out‚ 16.8798828125 network.sent‚ 3.820451e+06 disk.drbd1931.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.740223566691082 network.recv‚275627 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 cpu‚25.27697196453756 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚13699.30419921875 disk.sda.in‚0.03125 disk.sda2.out‚ 108.63671875# disk.dm-0.out‚13699.30419921875 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 59.69140625 disk.drbd1584.out‚0 disk.drbd1673.in‚0$ memory_percent‚91.14430745442709% disk./.usageGB‚1260.8521537780762 disk.drbd1584.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0$ proc.memory.availableMB‚2176.375( disk./.usagePercent‚70.5134529391742 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 59.69140625 disk.sda1.in‚0 disk.sda2.in‚0.03125ÄáT­ŸÒ ‚(:” ¼—ŠÓ¸Ö°³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚318.263' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚106.08766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶@Ö DÓ ‚(:¹  Ë—ŠÓ¿èг disk.nvme0n1.in‚ 59.69140625 disk.drbd1550.in‚0 disk.drbd1226.out‚0' disk./.usagePercent‚70.513451872485 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2111.out‚0 network.sent‚ 3.880577e+06 disk.drbd1931.in‚0 disk.rbd1.out‚ 16.9736328125 proc.cpu.threads‚17 network.recv‚277180# disk.dm-0.out‚13699.65966796875 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.sda.out‚ 109.00390625 disk.dm-0.in‚ 59.69140625 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚13699.65966796875 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1942.31640625% disk./.usageGB‚1260.8521347045898 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 109.00390625 disk.drbd1356.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.sda1.in‚0 disk.drbd2041.in‚0# memory_percent‚92.0966943105062* proc.memory.percent‚11.740223566691082 cpu‚25.27693350808663 disk.sda1.out‚0 disk.drbd1673.out‚0Ì}ª2žÔ ‚(:“ Ë—ŠÓ•¼¬³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.434' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.81133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dFèÚÕ ‚(:’ Ú—ŠÓ¦’Á³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚322.212' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚107.40399999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïî\%@Ö ‚(:µ  Ú—ŠÓé¢è³ disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 109.27734375 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.nvme0n1.in‚ 60.515625 disk.drbd1584.out‚0 disk.drbd2118.in‚0' disk./.usagePercent‚70.513451872485 disk.rbd1.in‚0 disk.rbd1.out‚ 17.2158203125 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.sda2.out‚ 109.27734375& disk.nvme0n1.out‚13699.92822265625 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.740223566691082 disk.drbd1817.in‚0 disk.dm-0.in‚ 60.515625 proc.cpu.threads‚17 disk.drbd2111.in‚0' proc.memory.availableMB‚ 1942.328125 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0# disk.dm-0.out‚13699.92822265625 cpu‚25.276954986737444 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8521347045898 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 network.sent‚ 3.888901e+06 network.recv‚278422 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0$ memory_percent‚92.09664662679037 disk.drbd1285.in‚0 disk.drbd1226.out‚0IVa22× - _wandbruntime‚3998  _runtime‚3998-ÎÉØ Ã ' lr-AdamW/pg1‚0.00011156553885196279 ' lr-AdamW/pg2‚0.00011156553885196279 trainer/global_step‚2199 % _timestamp‚1.7848432350376444e+09 _runtime‚3998.394873206 _step‚1794]ÁjÎÙ È _step‚179 ' lr-AdamW/pg1‚0.00011156553885196279 ' lr-AdamW/pg2‚0.00011156553885196279 trainer/global_step‚2199 % _timestamp‚1.7848432350376444e+09 _runtime‚3998.394873206³Ë¢-L2Ú - _wandbruntime‚3998  _runtime‚3998C³ëò'Û ìO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633650954354306 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19037838 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚633904347424768 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176226222472704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631363703515440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176226648122368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2288620878120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19037832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633905719034368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 _runtime‚3998.396277239 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39437607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20399769 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚631613234128896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 _step‚180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20389705 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633904347424768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19037832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚631613260302336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802025659904 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2292458732032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824[Üï«_ Ü ñO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12541832 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6681262 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚633652324393560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2287276266834 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5860947 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚631613234128896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2292458732032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39427537 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20399769 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633904347424768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19037832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚631613260302336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802025659904 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2292458732032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633650954354306 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19037838 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freedzqzƘ‚633904347424768 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176226222472704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631363703515440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176226648122368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2288620878120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19037832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633905719034368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 _runtime‚3998.396277239 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39437607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20399769 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚631613234128896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 _step‚180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20389705 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12547414 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19046205 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2288297068474 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚633932146969088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2292135803904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631892169592832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚173503151269888 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39444986 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634185650840064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176306479956992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2803275330048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚2199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39455070 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19046199 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚634185650840064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2292135803904 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39444986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631892143419392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚631892143419392 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6683827 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2289641679872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39455070 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚631642505289216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2293481247232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚633930776929722 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5863587 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20408865 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20398787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2293481247232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5863592 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20408865 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803328687104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19046205 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12547414 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19046205 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2288297068474 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚633932146969088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2292135803904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631892169592832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚173503151269888 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39444986 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634185650840064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176306479956992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2803275330048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚2199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39455070 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19046199 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚634185650840064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2292135803904 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39444986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631892143419392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚631892143419392 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6683827 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2289641679872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39455070 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚631642505289216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2293481247232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚633930776929722 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5863587 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20408865 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20398787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2293481247232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5863592 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20408865 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803328687104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653µ¤þÚH2à - _wandbruntime‚3999  _runtime‚3999<ù¶á û /loss/FRt2/detection_loss‚1.5710036212478184 .loss/FBt1/detection_loss‚0.706674637272954 epoch‚7 &loss/total_loss‚10.910709577601216 0loss/FBt1/assignment_loss‚0.4884848864894593 1loss/FBt2/assignment_loss‚0.17079877432727314 1loss/SRqqt1/detection_loss‚1.1555392431107618 trainer/global_step‚2199 _runtime‚3999.912574099 0loss/SRqqt2/detection_loss‚0.706282785575217 % _timestamp‚1.7848432365553734e+09 _step‚182 .loss/FBt2/detection_loss‚0.656230487744992 /loss/FRt1/assignment_loss‚2.500395033133806 0loss/FRt2/assignment_loss‚1.0433630605182305 2loss/SRqqt1/assignment_loss‚0.6911557531017157 2loss/SRqqt2/assignment_loss‚0.6204105980264071 -loss/FRt1/detection_loss‚0.60037069705258Ëóâ € &loss/total_loss‚10.910709577601216 0loss/FBt1/assignment_loss‚0.4884848864894593 1loss/FBt2/assignment_loss‚0.17079877432727314 1loss/SRqqt1/detection_loss‚1.1555392431107618 trainer/global_step‚2199 _runtime‚3999.912574099 0loss/SRqqt2/detection_loss‚0.706282785575217 % _timestamp‚1.7848432365553734e+09 _step‚182 .loss/FBt2/detection_loss‚0.656230487744992 /loss/FRt1/assignment_loss‚2.500395033133806 0loss/FRt2/assignment_loss‚1.0433630605182305 2loss/SRqqt1/assignment_loss‚0.6911557531017157 2loss/SRqqt2/assignment_loss‚0.6204105980264071 -loss/FRt1/detection_loss‚0.60037069705258 /loss/FRt2/detection_loss‚1.5710036212478184 .loss/FBt1/detection_loss‚0.706674637272954 epoch‚7¶Ò¯Átžã ‚(:“ é—ŠÓéËÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.675' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.55833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ7ßd@ä ‚(:µ  é—ŠÓù–ƒ´ disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 network.sent‚ 3.888967e+06 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 network.recv‚278488 disk.drbd1285.out‚0 disk.dm-0.in‚ 60.515625 proc.cpu.threads‚17% disk./.usageGB‚1260.8521347045898 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚13700.12744140625' disk./.usagePercent‚70.513451872485 disk.nvme0n1.in‚ 60.515625 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 17.2158203125 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.740223566691082 cpu‚25.276852140423713 disk.drbd1510.out‚0 disk.sda.out‚ 109.6328125 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 109.6328125 disk.drbd1285.in‚0 disk.drbd1938.in‚0$ memory_percent‚92.09787050882974 disk.drbd2111.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125& disk.nvme0n1.out‚13700.12744140625 disk.drbd2078.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1942.02734375 disk.drbd1673.in‚0vŽXå ‚(:’ ø—ŠÓÐýų gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚285.03' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.00999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0},«ð6æ ‚(:«  ø—ŠÓŠã³ network.sent‚ 3.892e+06 disk.dm-0.out‚13700.52734375 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 60.515625 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17$ memory_percent‚92.09802945454916 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.rbd1.out‚ 17.248046875 disk.drbd2111.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 109.8515625 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.sda.in‚0.03125 network.recv‚279618 disk.drbd1552.in‚0' disk./.usagePercent‚70.513451872485% disk./.usageGB‚1260.8521347045898 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1941.98828125 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.sda2.out‚ 109.8515625# proc.memory.rssMB‚ 2885.27734375# disk.nvme0n1.out‚13700.52734375 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 60.515625* proc.memory.percent‚11.740223566691082 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 cpu‚25.276817669571876Ð÷žç ‚(:“ ‡˜ŠÓߗų gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚193.567' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.52233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BSì7è ‚(:¬  ‡˜ŠÓ¡Ûç³ disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2885.27734375 disk.drbd1584.in‚0 disk.drbd1828.out‚0" disk.nvme0n1.out‚ 13700.7421875# memory_percent‚92.0980135599772 disk.sda1.out‚0 disk.sda.out‚ 110.2421875 disk.drbd2078.out‚0 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1941.9921875 cpu‚25.276841203384926 disk.sda2.out‚ 110.2421875 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.dm-0.out‚ 13700.7421875 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8537673950195 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0 network.recv‚279684 disk.dm-0.in‚ 60.515625 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 17.341796875 disk.nvme0n1.in‚ 60.515625 disk.drbd1226.out‚0 disk.drbd1510.out‚0 network.sent‚ 3.892136e+06 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.740223566691082) disk./.usagePercent‚70.51354318108089 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1931.in‚0Iâ‚ÍŸé ‚(:” –˜ŠÓ’ùÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.876' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.29199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è.²^7ê ‚(:¬  –˜ŠÓåûò³ disk.drbd1931.out‚0 disk.drbd1828.in‚0 network.sent‚ 3.955025e+06 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8542289733887 disk.drbd1289.in‚0 disk.dm-0.out‚13701.04296875) disk./.usagePercent‚70.51356899495963( proc.memory.availableMB‚ 1941.9609375 cpu‚25.27674146093005 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0# disk.nvme0n1.out‚13701.04296875# proc.memory.rssMB‚ 2885.27734375 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 60.515625 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0$ memory_percent‚92.09814071655273 disk.drbd1552.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 network.recv‚281733 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 110.484375 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.rbd1.out‚ 17.404296875 disk.drbd1817.out‚0 disk.dm-0.in‚ 60.515625* proc.memory.percent‚11.740223566691082 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.sda.out‚ 110.484375 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0·œúø9ë ‚(:®  ¥˜ŠÓ§´Ó³ disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.740223566691082 disk.drbd1584.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.dm-0.out‚13701.21484375 network.sent‚ 3.955091e+06 disk.drbd1673.in‚0 disk.drbd1931.out‚0 cpu‚25.276768786574735 disk.drbd1552.out‚0 disk.sda2.out‚ 110.7890625 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0# disk./.usageGB‚1260.84965133667 disk.drbd1673.out‚0 disk.drbd1938.in‚0# disk.nvme0n1.out‚13701.21484375 disk.drbd2118.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1942.20703125 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 110.7890625 disk.sda1.out‚0 disk.nvme0n1.in‚ 60.515625 disk.drbd1226.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51331298955061 disk.sda2.in‚0.03125 disk.dm-0.in‚ 60.515625 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 17.404296875 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0$ memory_percent‚92.09713935852051# proc.memory.rssMB‚ 2885.27734375 network.recv‚281799 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.rbd1.in‚0Øþ`ü”ì ‚(:‰ ¥˜ŠÓú£¯³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.914' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 r' ”í ‚(:‰ ´˜ŠÓ°Š¼³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.225' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.075 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kAG;î ‚(:°  ´˜ŠÓ‰Ûس disk.drbd2078.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 cpu‚25.282748040828576 disk.drbd2111.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2885.27734375* proc.memory.percent‚11.740223566691082 network.sent‚ 3.955157e+06) disk./.usagePercent‚70.51331298955061 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0$ memory_percent‚92.09814071655273 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 111.11328125 disk.drbd1356.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1941.9609375 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17# disk./.usageGB‚1260.84965133667 disk.dm-0.in‚ 60.515625 disk.drbd2111.in‚0# disk.nvme0n1.out‚13701.58203125 disk.drbd1510.out‚0 network.recv‚281865 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 111.11328125 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.dm-0.out‚13701.58203125 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 17.4365234375 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 60.515625 disk.drbd1817.in‚0Æ9B?2ï - _wandbruntime‚4089  _runtime‚4089,‹ö‘;ð ‹w >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2647010087966919 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9963292479515076 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6311240196228027 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14895500242710114 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982143044471741 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42680516839027405 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7821944355964661 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.544934868812561 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.564943253993988 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1933843344449997 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1805555671453476 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047109443694353104 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.802683413028717 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21712075173854828 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9973686337471008 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12693415582180023 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11445260047912598 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5813336968421936 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19205836951732635 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6295346617698669 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8888100981712341 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.777845561504364 trainer/global_step‚2239 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975681304931641 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6552054286003113 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9749802947044373 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982041716575623 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5406659841537476 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9953917264938354 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008920698426663876 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7571438550949097 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5046989321708679 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17979808151721954 /particle/accuracy_4_of_4‚0.9410868287086487 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7590059041976929 +"Purity/2FRt0SRqqt2FBt/event_purity‚0.75 +jet/accuracy_1_of_1‚0.31511157751083374 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7272727489471436 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17992259562015533 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7466424703598022 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ,jet/accuracy_3_of_3‚0.050967372953891754 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2024175375699997 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9957574605941772 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11448778212070465 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11277033388614655 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017820170149207115 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7948756217956543 +jet/accuracy_2_of_2‚0.13901135325431824 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14568521082401276 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09949769824743271 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37027424573898315 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9948440790176392 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5370491147041321 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1466159075498581 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7447608709335327 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4415290951728821 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902281761169434 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3384615480899811 *jet/accuracy_1_of_3‚0.9205759167671204 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5034413933753967 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6154480576515198 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16773511469364166 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5881848931312561 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6291008591651917 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36057665944099426 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1871759444475174 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1814238280057907 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08058217912912369 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.668993353843689 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2170903980731964 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7929168939590454 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18479855358600616 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.553192675113678 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5631868243217468 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2040056437253952 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7935448884963989 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6722590327262878 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2040189951658249 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08058217912912369 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985713958740234 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_3‚0.5170594453811646 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1325836032629013 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09846575558185577 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6435933113098145 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6443303823471069 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7487167716026306 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.018756305798888206 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9826509356498718 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18474359810352325 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.992967426776886 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6251773238182068 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3354906737804413 +particle/sensitivity‚0.6187697052955627 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19624409079551697 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6251773238182068 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7272727489471436 _runtime‚4089.023232418 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795752167701721 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5151515007019043 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.574978768825531 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935064911842346 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2149008959531784 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6638607382774353 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24824786186218262 /particle/accuracy_3_of_3‚0.9985468983650208 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8483384251594543 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23356054723262787 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8852406740188599 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8074005246162415 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804563522338867 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5834891200065613 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9979591965675354 +particle/specificity‚0.9051916003227234 % _timestamp‚1.7848433256651561e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9836882948875427 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.341952383518219 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43067964911460876 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.629487156867981 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8888888359069824 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8570235371589661 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21822556853294373 *loss/val_total_loss‚10.970086258479398 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1805555671453476 6validation_average_jet_accuracy‚0.3578151762485504 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9833060503005981 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.300181120634079 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.023544637486338615 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2651973068714142 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7159899473190308 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.868055522441864 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6178616285324097 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7277513146400452 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6757296919822693 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7309511303901672 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.67423015832901 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6521595120429993 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04811004176735878 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 (particle/accuracy‚0.8437383770942688 _step‚183 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985713958740234 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15872101485729218 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 *jet/accuracy_1_of_2‚0.7779873013496399 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6735065579414368 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6119431853294373 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957922697067261 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.802683413028717 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47626590728759766 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2040056437253952 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚7 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37152451276779175 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5972200036048889 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04251944646239281 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9921132922172546 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6795514822006226 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6181246638298035 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7483159899711609 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.1051706075668335 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9930424690246582/7$µ©$ñ w 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6251773238182068 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7272727489471436 _runtime‚4089.023232418 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795752167701721 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5151515007019043 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.574978768825531 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935064911842346 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2149008959531784 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6638607382774353 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24824786186218262 /particle/accuracy_3_of_3‚0.9985468983650208 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8483384251594543 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23356054723262787 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8852406740188599 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8074005246162415 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804563522338867 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5834891200065613 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9979591965675354 +particle/specificity‚0.9051916003227234 % _timestamp‚1.7848433256651561e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9836882948875427 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.341952383518219 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43067964911460876 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.629487156867981 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8888888359069824 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8570235371589661 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21822556853294373 *loss/val_total_loss‚10.970086258479398 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1805555671453476 6validation_average_jet_accuracy‚0.3578151762485504 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9833060503005981 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.300181120634079 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.023544637486338615 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2651973068714142 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7159899473190308 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.868055522441864 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6178616285324097 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7277513146400452 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6757296919822693 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7309511303901672 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.67423015832901 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6521595120429993 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04811004176735878 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 (particle/accuracy‚0.8437383770942688 _step‚183 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985713958740234 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15872101485729218 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 *jet/accuracy_1_of_2‚0.7779873013496399 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6735065579414368 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6119431853294373 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957922697067261 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.802683413028717 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47626590728759766 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2040056437253952 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚7 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37152451276779175 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5972200036048889 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04251944646239281 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9921132922172546 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6795514822006226 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6181246638298035 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7483159899711609 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.1051706075668335 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9930424690246582 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2647010087966919 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9963292479515076 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6311240196228027 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14895500242710114 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982143044471741 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42680516839027405 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7821944355964661 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.544934868812561 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.564943253993988 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1933843344449997 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1805555671453476 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047109443694353104 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.802683413028717 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21712075173854828 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9973686337471008 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12693415582180023 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11445260047912598 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5813336968421936 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19205836951732635 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6295346617698669 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8888100981712341 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.777845561504364 trainer/global_step‚2239 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975681304931641 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6552054286003113 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9749802947044373 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982041716575623 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5406659841537476 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9953917264938354 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008920698426663876 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7571438550949097 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5046989321708679 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17979808151721954 /particle/accuracy_4_of_4‚0.9410868287086487 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7590059041976929à`U`í +"Purity/2FRt0SRqqt2FBt/event_purity‚0.75 +jet/accuracy_1_of_1‚0.31511157751083374 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7272727489471436 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17992259562015533 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7466424703598022 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ,jet/accuracy_3_of_3‚0.050967372953891754 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2024175375699997 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9957574605941772 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11448778212070465 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11277033388614655 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017820170149207115 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7948756217956543 +jet/accuracy_2_of_2‚0.13901135325431824 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14568521082401276 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09949769824743271 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37027424573898315 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9948440790176392 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5370491147041321 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1466159075498581 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7447608709335327 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4415290951728821 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902281761169434 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3384615480899811 *jet/accuracy_1_of_3‚0.9205759167671204 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5034413933753967 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6154480576515198 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16773511469364166 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5881848931312561 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6291008591651917 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36057665944099426 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1871759444475174 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1814238280057907 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08058217912912369 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.668993353843689 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2170903980731964 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7929168939590454 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18479855358600616 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.553192675113678 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5631868243217468 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2040056437253952 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7935448884963989 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6722590327262878 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2040189951658249 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08058217912912369 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985713958740234 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_3‚0.5170594453811646 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1325836032629013 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09846575558185577 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6435933113098145 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6443303823471069 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7487167716026306 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.018756305798888206 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9826509356498718 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18474359810352325 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.992967426776886 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6251773238182068 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3354906737804413 +particle/sensitivity‚0.6187697052955627 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19624409079551697 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0·tî_û“ò ‚(:ˆ ØŠÓÃÇͳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚289.038' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç¢‚-ó ‚(:¢  ØŠÓ…Êì³ disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0$ memory_percent‚88.14659118652344 cpu‚25.285230813456224 disk.drbd1584.in‚0 disk.sda.out‚111.5625 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 60.515625 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 network.sent‚ 3.960845e+06 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 17.5302734375& proc.memory.availableMB‚ 2913.09375! disk.nvme0n1.out‚ 14671.328125) disk./.usagePercent‚70.51354019435111 disk.drbd1938.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.742337544759113 network.recv‚282845 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 60.515625! proc.memory.rssMB‚ 2885.796875 proc.cpu.threads‚17% disk./.usageGB‚1260.8537139892578 disk.drbd2118.out‚0 disk.dm-0.out‚ 14671.328125 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚111.5625 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1673.in‚0XrB·2ô - _wandbruntime‚4109  _runtime‚4109¤ aQÇõ Á & lr-AdamW/pg1‚0.0001112768355448033 & lr-AdamW/pg2‚0.0001112768355448033 trainer/global_step‚2249 % _timestamp‚1.7848433464908948e+09 _runtime‚4109.848155144 _step‚184 ¼¿¥Ìö Æ trainer/global_step‚2249 % _timestamp‚1.7848433464908948e+09 _runtime‚4109.848155144 _step‚184 & lr-AdamW/pg1‚0.0001112768355448033 & lr-AdamW/pg2‚0.0001112768355448033¸,Ûý2÷ - _wandbruntime‚4109  _runtime‚4109 üüó'ø íO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20903033 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12853418 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚647862767927440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2343988988956 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19545614 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5999839 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚647862793355408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20892969 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6853579 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177402882714624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650467296669696 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚648119401551872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2349266727424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚648119401551872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12853667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20892969 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180273876807680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚650468668279296 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚648119375378432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19545614 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5999844 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6853823 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650468668279296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 % _timestamp‚1.7848433464919891e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248…76ø'ù òO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚647862793355408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20892969 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6853579 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177402882714624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650467296669696 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚648119401551872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2349266727424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚648119401551872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12853667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20892969 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180273876807680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚650468668279296 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚648119375378432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19545614 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5999844 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6853823 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650468668279296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 % _timestamp‚1.7848433464919891e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20903033 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12853418 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚647862767927440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2343988988956 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19545614 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5999839¹ü*<-Ÿú ‚(:” Ò˜ŠÓ’ú¼³ gpu.0.gpu‚94.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚335.465' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚111.82166666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü ?û ‚(:´  Ò˜ŠÓËã³ disk.drbd1226.out‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 111.83203125 disk.drbd1289.in‚0 disk.rbd1.out‚ 17.5927734375 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 60.515625 cpu‚25.285164528943504! proc.memory.rssMB‚ 2885.796875 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 60.515625 disk.drbd1584.in‚0 disk.sda.in‚0.03125* proc.memory.percent‚11.742337544759113% disk.nvme0n1.out‚14671.9892578125 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0% disk./.usageGB‚1260.8537559509277 disk.drbd2118.in‚0$ memory_percent‚88.14751307169595 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 network.recv‚282953 disk.drbd1828.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 111.83203125 disk.drbd1938.in‚0" disk.dm-0.out‚14671.9892578125 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 network.sent‚ 3.960953e+06) disk./.usagePercent‚70.51354254106737 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0( proc.memory.availableMB‚ 2912.8671875 proc.cpu.threads‚17ód2ü - _wandbruntime‚4111  _runtime‚4111úuàÀý ÷M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40456032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20912129 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6856589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177481469080576 @:DeviceStatsMonitor.on_train_batch_end/num_syncŠzi9÷_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648141569701216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2345009790596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20912129 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2350289242624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6856844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6002479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180353777594368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40456032 _runtime‚4111.330148219 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20902051 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚648398310842368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚650487949531178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19553981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40466116 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2348943799296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650748600084992 % _timestamp‚1.7848433479724967e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650747228468224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2872308513792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚648398310842368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12859328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12859068 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648141595129184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19553981 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2872274031104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40466116 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚650486579491812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚648398284668928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2348943799296 _step‚186 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180353375020544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177481100989440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19553987 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚2249 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2350289242624 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2348943799296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650748600084992 % _timestamp‚1.7848433479724967e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650747228468224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2872308513792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚648398310842368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12859328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12859068 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648141595129184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19553981 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2872274031104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40466116 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚650486579491812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚648398284668928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2348943799296 _step‚186 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180353375020544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177481100989440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19553987 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚2249 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2350289242624 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20912129 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6856589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177481469080576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648141569701216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2345009790596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20912129 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2350289242624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6856844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6002479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180353777594368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40456032 _runtime‚4111.330148219 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20902051 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚648398310842368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚650487949531178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19553981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40466116ºm„ýI2ÿ - _wandbruntime‚4111  _runtime‚4111ÇŠ€  1loss/FBt2/assignment_loss‚0.14705090470165547 /loss/FBt1/detection_loss‚1.2987906757742167 &loss/total_loss‚11.468343690537402 0loss/FRt2/assignment_loss‚1.0453960191610172 2loss/SRqqt2/assignment_loss‚0.6331639420873373 epoch‚8 0loss/FRt1/assignment_loss‚2.5708388051351867 2loss/SRqqt1/assignment_loss‚0.7372810207069409 /loss/FRt1/detection_loss‚0.5850533457046764 /loss/FRt2/detection_loss‚1.5445731523556305 1loss/SRqqt1/detection_loss‚1.3090874546778517 1loss/SRqqt2/detection_loss‚0.6711013792546723 % _timestamp‚1.7848433480068793e+09 _step‚187 0loss/FBt1/assignment_loss‚0.2000706815189915 /loss/FBt2/detection_loss‚0.7259363094592257 trainer/global_step‚2249 _runtime‚4111.364095096¦)ð  † 0loss/FRt2/assignment_loss‚1.0453960191610172 2loss/SRqqt2/assignment_loss‚0.6331639420873373 epoch‚8 0loss/FRt1/assignment_loss‚2.5708388051351867 2loss/SRqqt1/assignment_loss‚0.7372810207069409 /loss/FRt1/detection_loss‚0.5850533457046764 /loss/FRt2/detection_loss‚1.5445731523556305 1loss/SRqqt1/detection_loss‚1.3090874546778517 1loss/SRqqt2/detection_loss‚0.6711013792546723 % _timestamp‚1.7848433480068793e+09 _step‚187 0loss/FBt1/assignment_loss‚0.2000706815189915 /loss/FBt2/detection_loss‚0.7259363094592257 trainer/global_step‚2249 _runtime‚4111.364095096 1loss/FBt2/assignment_loss‚0.14705090470165547 /loss/FBt1/detection_loss‚1.2987906757742167 &loss/total_loss‚11.468343690537402»‘ö=‚ ‚(:²  ᘊÓÖ‰‡´ disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0# memory_percent‚88.1479263305664! proc.memory.rssMB‚ 2885.796875 disk.drbd1938.in‚0" disk.dm-0.out‚15112.1103515625 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 112.26953125 disk.drbd1931.in‚0 disk.drbd1931.out‚0% disk.nvme0n1.out‚15112.4892578125 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 cpu‚25.285094229191085 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0' proc.memory.availableMB‚ 2912.765625 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.8498992919922 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 60.515625 disk.rbd1.out‚ 17.5927734375 disk.drbd1510.out‚0* proc.memory.percent‚11.742337544759113 disk.drbd1226.out‚0 disk.drbd1510.in‚0 network.sent‚ 3.961019e+06 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 60.515625 network.recv‚283019 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 112.26953125 disk.drbd2078.out‚0 disk.drbd1584.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51332685651026 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 Ïžƒ ‚(:“ ᘊÓ€Ôâ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚294.218' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.07266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùxc⟄ ‚(:” ð˜ŠÓÄÏë³ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚67.0 gpu.0.powerWatts‚303.721' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.24033333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÜØX5… ‚(:ª  ð˜ŠÓÙ­‹´ disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.rbd1.out‚17.625 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0$ memory_percent‚88.14878463745117 network.recv‚283085) disk./.usagePercent‚70.51332685651026 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.dm-0.in‚ 60.515625 disk.drbd1552.in‚0 disk.sda.out‚ 112.5703125 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0" disk.dm-0.out‚15642.3076171875* proc.memory.percent‚11.742337544759113 cpu‚25.28496771981968% disk./.usageGB‚1260.8498992919922 disk.drbd1938.in‚0! proc.memory.rssMB‚ 2885.796875 network.sent‚ 3.961085e+06 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 60.515625% disk.nvme0n1.out‚15642.3076171875 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 112.5703125 disk.drbd1285.in‚0( proc.memory.availableMB‚ 2912.5546875 disk.drbd1673.out‚0ܥކ ‚(:“ ÿ˜ŠÓíÞѳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚68.0 gpu.0.powerWatts‚296.642' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.88066666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k`ó9‡ ‚(:®  ÿ˜ŠÓÈŠû³ disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1828.in‚0% disk./.usageGB‚1260.8499183654785 disk.drbd2111.in‚0" disk.dm-0.out‚15642.6318359375 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd2118.out‚0 disk.sda.out‚ 112.94921875 disk.drbd1938.out‚0 network.recv‚285021 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 2912.5546875 network.sent‚ 4.020326e+06 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.dm-0.in‚ 60.515625 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51332792319947 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.742337544759113 disk.sda2.out‚ 112.94921875! proc.memory.rssMB‚ 2885.796875 disk.drbd1356.out‚0% disk.nvme0n1.out‚15642.6318359375 disk.drbd2118.in‚0 disk.rbd1.out‚17.71875 disk.sda1.out‚0 disk.drbd1289.out‚0 cpu‚25.28484410123453 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 60.515625 disk.drbd1673.out‚0$ memory_percent‚88.14878463745117 disk.drbd1552.in‚0 disk.drbd1356.in‚0ðc|lˆ ‚(:’ Ž™ŠÓçéÖ³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚67.0 gpu.0.powerWatts‚267.66' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.22000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gíëÇ8‰ ‚(:­  Ž™ŠÓáÑõ³ disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 network.sent‚ 4.020392e+06 disk.drbd1931.in‚0" disk.dm-0.out‚15643.2197265625 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0$ memory_percent‚88.14878463745117 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 cpu‚25.284839260210827 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0) disk./.usagePercent‚70.51332685651026% disk./.usageGB‚1260.8498992919922 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚17.78125 disk.sda.out‚ 113.2421875 disk.drbd2111.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 2912.5546875 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 113.2421875 disk.nvme0n1.in‚ 60.515625 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚15643.2197265625* proc.memory.percent‚11.742337544759113 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 60.515625 disk.drbd1550.out‚0 network.recv‚285087 disk.sda2.in‚0.03125 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0ŸP•ïœŠ ‚(:‘ ™ŠÓ”Æͳ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚200.599' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.86633333333333 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/òÑ=‹ ‚(:²  ™ŠÓËÍê³ disk.drbd1931.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51332792319947% disk./.usageGB‚1260.8499183654785 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 62.01171875% disk.nvme0n1.out‚15643.3486328125 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 62.01171875 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 113.52734375 disk.drbd1931.in‚0 disk.drbd1289.in‚0 network.sent‚ 4.088148e+06 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 cpu‚25.28477567726719 disk.drbd1584.in‚0 disk.drbd1938.out‚0" disk.dm-0.out‚15643.3486328125 disk.drbd2111.out‚0 disk.rbd1.out‚17.78125( proc.memory.availableMB‚ 2912.5546875 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0! proc.memory.rssMB‚ 2885.796875$ memory_percent‚88.14878463745117* proc.memory.percent‚11.742337544759113 network.recv‚286771 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 113.52734375 disk.sda2.in‚0.03125 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0ª½cÃ2Œ - _wandbruntime‚4197  _runtime‚4197VÕÈÉ Ã trainer/global_step‚2299 % _timestamp‚1.7848434336807995e+09 _runtime‚4197.038039665 _step‚188 ' lr-AdamW/pg1‚0.00011098122285966333 ' lr-AdamW/pg2‚0.00011098122285966333dÛëÎŽ È ' lr-AdamW/pg1‚0.00011098122285966333 ' lr-AdamW/pg2‚0.00011098122285966333 trainer/global_step‚2299 % _timestamp‚1.7848434336807995e+09 _runtime‚4197.038039665 _step‚188¼ÞÊú‰2 - _wandbruntime‚4197  _runtime‚4197ºÖ5  íO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21358985 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2395092871764 % _timestamp‚1.7848434336818693e+09 _step‚189 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184266486354944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2934285342720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2400456283648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21348921 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚661802856616240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.c¥Qìurrent‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664463950750720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚662064866076672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6131844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2399110847488 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181331832921088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7002644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184266069093888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚664199319527258 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚662064866076672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13134825 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41312885 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131839 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181332201012224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2934236172800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚661802882044208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664465322360320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41322955 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19963970 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664463950750720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19963964 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚662064839903232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7002981 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚664197949488004 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚2299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664463950750720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19963964 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚662064839903232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7002981 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚664197949488004 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚2299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21358985 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2395092871764 % _timestamp‚1.7848434336818693e+09 _step‚189 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184266486354944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2934285342720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2400456283648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21348921 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚661802856616240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664463950750720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚662064866076672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6131844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2399110847488 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181331832921088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7002644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184266069093888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚664199319527258 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚662064866076672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13134825 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41312885 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131839 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181332201012224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248½‘)ƒ2’ - _wandbruntime‚4198  _runtime‚4198É$¬±ü&“ öM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664745254166016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184346322539520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2396113673404 $ _timestamp‚1.784843435160019e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184345896897024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚664477772063420 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41340418 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6134479 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7005922 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662081658390016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2397458284802 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2401478798848 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13140406 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664745254166016 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4198.517631921 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21368081 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2400133355520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41330334 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13140063 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2401478798848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21368081 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7005584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19972331 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚662343749193728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41340418 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41330334 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19972331 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2400133355520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2935477609984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚662343775367168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚662343775367168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚662343749193728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6134484 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚6620816838179845¾ÆùË” ûM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664745254166016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184346322539520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2396113673404 $ _timestamp‚1.784843435160019e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184345896897024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚664477772063420 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41340418 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6134479 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7005922 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662081658390016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2397458284802 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2401478798848 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13140406 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664745254166016 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4198.517631921 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21368081 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2400133355520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41330334 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13140063 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2401478798848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21368081 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7005584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19972331 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 163rºU680172544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚662343749193728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41340418 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41330334 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19972331 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2400133355520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2935477609984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚662343775367168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚662343775367168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚662343749193728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6134484 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662081683817984¾jNC™2• - _wandbruntime‚4198  _runtime‚4198º À– ÿ /loss/FRt1/detection_loss‚0.6020528918903646 &loss/total_loss‚11.237836362477381 % _timestamp‚1.7848434351968303e+09 2loss/SRqqt1/assignment_loss‚0.7287909612024182 2loss/SRqqt2/assignment_loss‚0.6583626347120038 1loss/SRqqt1/detection_loss‚1.1427961362175247 /loss/FBt2/detection_loss‚0.7643755991247569 epoch‚8 _step‚191 /loss/FRt2/assignment_loss‚1.026329024841911 1loss/FBt2/assignment_loss‚0.10641065518920191 .loss/FRt2/detection_loss‚1.607964882729426 1loss/SRqqt2/detection_loss‚0.7326589105226721 trainer/global_step‚2299 0loss/FRt1/assignment_loss‚2.5630815799726823 0loss/FBt1/assignment_loss‚0.3859368091313324 /loss/FBt1/detection_loss‚0.9190762769430876 _runtime‚4198.554077171݆j­ — „ /loss/FRt2/assignment_loss‚1.026329024841911 1loss/FBt2/assignment_loss‚0.10641065518920191 .loss/FRt2/detection_loss‚1.607964882729426 1loss/SRqqt2/detection_loss‚0.7326589105226721 trainer/global_step‚2299 0loss/FRt1/assignment_loss‚2.5630815799726823 0loss/FBt1/assignment_loss‚0.3859368091313324 /loss/FBt1/detection_loss‚0.9190762769430876 _runtime‚4198.554077171 /loss/FRt1/detection_loss‚0.6020528918903646 &loss/total_loss‚11.237836362477381 % _timestamp‚1.7848434351968303e+09 2loss/SRqqt1/assignment_loss‚0.7287909612024182 2loss/SRqqt2/assignment_loss‚0.6583626347120038 1loss/SRqqt1/detection_loss‚1.1427961362175247 /loss/FBt2/detection_loss‚0.7643755991247569 epoch‚8 _step‚191¿ÌHx垘 ‚(:“ ¬™ŠÓ¼¯Î³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚297.984' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.32799999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UGèF™ ‚(:»  ¬™ŠÓ˜äð³ disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 113.91015625 disk.sda2.out‚ 113.91015625 disk.sda.in‚0.03125 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.742337544759113 network.recv‚286837 disk.dm-0.in‚ 62.01171875 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 17.8134765625! proc.memory.rssMB‚ 2885.796875 disk.drbd1285.in‚0 disk.drbd2078.in‚0) proc.memory.availableMB‚ 2912.70703125 disk.drbd1673.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd2078.out‚0 cpu‚25.284710922867266 network.sent‚ 4.088214e+06& disk.nvme0n1.out‚15643.93701171875# disk.dm-0.out‚15643.93701171875% disk./.usageGB‚1260.8495330810547 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0$ memory_percent‚88.14816474914551) disk./.usagePercent‚70.51330637607755 disk.sda1.out‚0{Ô‘ÛCš ‚(:¸  »™ŠÓÙþ´ disk.drbd1552.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 62.01171875) proc.memory.availableMB‚ 2912.45703125& disk.nvme0n1.out‚15644.19482421875 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 114.3203125 disk.drbd2078.in‚0* proc.memory.percent‚11.742337544759113 disk.sda.out‚ 114.3203125 disk.drbd1931.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚15644.19482421875 cpu‚25.28464557249704 proc.cpu.threads‚17 disk.rbd1.out‚ 17.9072265625 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 network.recv‚286945% disk./.usageGB‚1260.8495330810547 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1817.in‚0 disk.drbd1356.in‚0$ memory_percent‚88.14918200174967 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 network.sent‚ 4.088322e+06 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0) disk./.usagePercent‚70.51330637607755 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.in‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd1938.out‚0…BÝž› ‚(:“ »™ŠÓ¡Ðã³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚237.836' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚79.27866666666668 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J,£Bœ ‚(:·  Ê™ŠÓô§ø³ disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 2912.46484375 disk.sda2.in‚0.03125 disk.rbd1.in‚0 cpu‚25.284641855927646 network.recv‚287897) disk./.usagePercent‚70.51330658941538 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 network.sent‚ 4.093842e+06 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚15644.60888671875 disk.drbd1550.out‚0 disk.sda2.out‚ 114.6796875 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 62.01171875! proc.memory.rssMB‚ 2885.796875 disk.drbd1584.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.742337544759113# disk.dm-0.out‚15644.60888671875 disk.sda.out‚ 114.6796875 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 17.9697265625# memory_percent‚88.1491502126058 disk.drbd2118.in‚0 disk.dm-0.in‚ 62.01171875$ disk./.usageGB‚1260.849536895752 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.sda.in‚0.03125 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0’°Ÿ ‚(:” Ê™ŠÓÝäÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚110.026' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.675333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Êtžž ‚(:“ Ù™ŠÓ °ï³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.445' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.48166666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¥CŸ ‚(:¸  Ù™ŠÓŒÿŽ´ disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.rbd1.out‚ 17.9697265625 disk.nvme0n1.in‚ 62.01171875 disk.drbd1510.in‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚15644.77294921875 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.742337544759113 disk.drbd2111.in‚0 disk.drbd2118.in‚0$ memory_percent‚88.15024693806967( proc.memory.availableMB‚ 2912.1953125 network.sent‚ 4.09395e+06$ disk./.usageGB‚1260.849536895752 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd1828.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51330658941538 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1584.in‚0 network.recv‚288005 disk.sda.in‚0.03125 disk.sda2.out‚ 115.04296875 disk.drbd1289.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1584.out‚0 cpu‚25.284518205528606 disk.drbd1356.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚15644.77294921875 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 115.04296875G!‚q“  ‚(:ˆ 虊Óƒ—̳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚286.899' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.633 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ÊîÜA¡ ‚(:¶  虊Ó»Àê³ disk.drbd1510.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1965.out‚0 disk.drbd2041.out‚0 network.sent‚ 4.094016e+06% disk./.usageGB‚1260.8495559692383 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1226.in‚0 cpu‚25.284459398521356! proc.memory.rssMB‚ 2885.796875 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1550.out‚0 network.recv‚288071 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 115.28515625 disk.drbd1550.in‚0) disk./.usagePercent‚70.51330765610459 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0! disk.dm-0.out‚15645.083984375 disk.rbd1.out‚ 18.1181640625$ disk.nvme0n1.out‚15645.083984375 disk.drbd1552.in‚0 disk.drbd1965.in‚0$ memory_percent‚88.15043767293295( proc.memory.availableMB‚ 2912.1484375 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 62.01171875 proc.cpu.threads‚17 disk.sda.out‚ 115.28515625* proc.memory.percent‚11.742337544759113 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0„©Už¢ ‚(:“ ÷™ŠÓŽ´Â³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚292.292' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.43066666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽüÁ=£ ‚(:²  ÷™ŠÓ‡«ã³ disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1938.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 115.69140625 disk.drbd1285.out‚0* proc.memory.percent‚11.742337544759113 disk.drbd1584.out‚0 disk.dm-0.in‚ 62.01171875 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0! proc.memory.rssMB‚ 2885.796875 network.sent‚ 4.159083e+06 disk.drbd1965.in‚0 disk.drbd2078.in‚0$ proc.memory.availableMB‚1940.125) disk./.usagePercent‚70.51330786944243 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 115.69140625 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8495597839355! disk.dm-0.out‚15645.291015625 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 62.01171875$ memory_percent‚92.10561116536459 disk.drbd1285.in‚0 disk.drbd1931.in‚0$ disk.nvme0n1.out‚15645.291015625 cpu‚25.284514156114938 network.recv‚289669 disk.drbd2118.out‚0 disk.rbd1.out‚ 18.7666015625 disk.drbd1584.in‚0 disk.drbd1828.in‚0¹ÓÕ2¤ - _wandbruntime‚4284  _runtime‚4284%þ¯É¥ à ' lr-AdamW/pg2‚0.00011067873954459302 trainer/global_step‚2349 % _timestamp‚1.7848435207002006e+09 _runtime‚4284.057427167 _step‚192 ' lr-AdamW/pg1‚0.00011067873954459302Oñ³…Φ È ' lr-AdamW/pg1‚0.00011067873954459302 ' lr-AdamW/pg2‚0.00011067873954459302 trainer/global_step‚2349 % _timestamp‚1.7848435207002006e+09 _runtime‚4284.057427167 _step‚192ÀͤLI2§ - _wandbruntime‚4284  _runtime‚4284óW áò'¨ ìO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42187187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2450300403712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185261519309824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2451645839872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21814937 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚676010330601472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676010304428032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13413957 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚193 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678461976441344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7150463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚675742970733008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2450300403712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188257797359104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21804873 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678460604831744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21814937 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2451645839872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20382320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678461976441344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21804873 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676010330601472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13414307 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188258220911616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 4284.05883827 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚350 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678460604831744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚678190512098866 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 % _timestamp‚1.7848435207012463e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248¬Ó²ƒÌ © ñO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚350 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678460604831744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚678190512098866 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 % _timestamp‚1.7848435207012463e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42187187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2450300403712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185261519309824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2451645839872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_po‹ü*À+ol.allocated‚21814937 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚676010330601472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676010304428032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13413957 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚193 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678461976441344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7150463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚675742970733008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2450300403712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188257797359104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21804873 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678460604831744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21814937 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2451645839872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20382320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678461976441344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21804873 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676010330601472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13414307 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188258220911616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 4284.05883827ÁŸ[`%2ª - _wandbruntime‚4285  _runtime‚4285âÇß‘ý&« ÷M I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2452668355072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2452668355072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚4285.538023195 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21813955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2349 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188337640914432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185339737584640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678740536630272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185340105675776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2447217556212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2448562167610 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20390687 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13419510 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2451322911744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678741908247040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678741908247040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7153377 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188338060265472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676021747078816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21824033 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚678470334674394 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚676021772506784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚676289239891968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42214720 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20390681 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚194 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181šIž'¬ üM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678741908247040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7153377 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188338060265472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676021747078816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21824033 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚678470334674394 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚676021772506784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚676289239891968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42214720 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20390681 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚194 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2452668355072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2452668355072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚4285.538023195 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21813955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2349 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188337640914432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185339737584640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678740536630272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185340105675776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2447217556212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2448562167610 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20390687 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13419510 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2451322911744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678741908247040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Âdš ?2­ - _wandbruntime‚4285  _runtime‚4285SiÒý® € /loss/FRt2/detection_loss‚1.5750229558741142 trainer/global_step‚2349 _runtime‚4285.575411375 0loss/FRt2/assignment_loss‚0.9161024245300463 /loss/FBt1/detection_loss‚0.7032459236004136 /loss/FBt2/detection_loss‚0.8395389730374855 &loss/total_loss‚10.697044559219162 % _timestamp‚1.7848435222181895e+09 0loss/FRt1/assignment_loss‚2.3900412692015887 0loss/FBt1/assignment_loss‚0.3402172061416257 /loss/FRt1/detection_loss‚0.5957918019999237 1loss/SRqqt1/detection_loss‚1.1811476105699905 1loss/SRqqt2/detection_loss‚0.7127332374403327 epoch‚8 _step‚195 2loss/SRqqt2/assignment_loss‚0.6618929042247974 0loss/FBt2/assignment_loss‚0.0890011752068513 2loss/SRqqt1/assignment_loss‚0.6923090773919922hŸO ¯ … 2loss/SRqqt1/assignment_loss‚0.6923090773919922 /loss/FRt2/detection_loss‚1.5750229558741142 trainer/global_step‚2349 _runtime‚4285.575411375 0loss/FRt2/assignment_loss‚0.9161024245300463 /loss/FBt1/detection_loss‚0.7032459236004136 /loss/FBt2/detection_loss‚0.8395389730374855 &loss/total_loss‚10.697044559219162 % _timestamp‚1.7848435222181895e+09 0loss/FRt1/assignment_loss‚2.3900412692015887 0loss/FBt1/assignment_loss‚0.3402172061416257 /loss/FRt1/detection_loss‚0.5957918019999237 1loss/SRqqt1/detection_loss‚1.1811476105699905 1loss/SRqqt2/detection_loss‚0.7127332374403327 epoch‚8 _step‚195 2loss/SRqqt2/assignment_loss‚0.6618929042247974 0loss/FBt2/assignment_loss‚0.0890011752068513Ã]Êÿð“° ‚(:ˆ †šŠÓЈг gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚280.068' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö‚þø<± ‚(:±  †šŠÓÜÑí³! proc.memory.rssMB‚ 2885.796875 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0% disk./.usageGB‚1260.8495788574219 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 network.sent‚ 4.159191e+06 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0" disk.dm-0.out‚15645.9228515625 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1940.16015625 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda.out‚116.0625 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚116.0625 disk.drbd1931.in‚0 disk.drbd2041.out‚0% disk.nvme0n1.out‚15645.9228515625 cpu‚25.284451255595492 disk.drbd1356.in‚0 disk.rbd1.out‚ 18.8447265625 disk.sda1.in‚0$ memory_percent‚92.10546811421713 network.recv‚289777) disk./.usagePercent‚70.51330893613164 disk.rbd1.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd2111.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.742337544759113 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0Œ„ðcœ² ‚(:‘ •šŠÓ¦¤Â³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚109.397' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.46566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ•29©³ ‚(:¶  •šŠÓšÄß³ disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0$ memory_percent‚92.10551579793295 disk.dm-0.in‚ 62.01171875 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.sda2.out‚ 116.62109375 disk.drbd1938.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 network.sent‚ 4.167404e+06 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.742337544759113% disk./.usageGB‚1260.8495407104492" disk.dm-0.out‚15646.1142578125 disk.drbd2041.out‚0 disk.drbd1550.out‚0 cpu‚25.2844487360271 disk.rbd1.out‚ 19.4697265625 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 network.recv‚290818 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd151ÿ/ —˜0.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.sda.out‚ 116.62109375 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51330680275323% disk.nvme0n1.out‚15646.1142578125 disk.drbd1817.out‚0( proc.memory.availableMB‚ 1940.1484375‚'Ķ“´ ‚(:ˆ ¤šŠÓý“г gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚269.193' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯5!ÂDµ ‚(:¹  ¤šŠÓ’šñ³ network.recv‚290884 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd2111.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.10564295450845 disk.drbd1285.out‚0& disk.nvme0n1.out‚15646.55615234375* proc.memory.percent‚11.742337544759113% disk./.usageGB‚1260.8495407104492 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51330680275323 disk.sda.out‚ 116.98828125 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 network.sent‚ 4.16747e+06 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.sda2.out‚ 116.98828125 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚15646.55615234375 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 62.01171875! proc.memory.rssMB‚ 2885.796875 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0( proc.memory.availableMB‚ 1940.1171875 cpu‚25.284328393981557 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 19.7275390625 disk.drbd1285.in‚0 àÑu“¶ ‚(:ˆ ³šŠÓÁˆå³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚231.762' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚77.254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º!­å@· ‚(:µ  ³šŠÓ嘈´ disk.drbd1931.out‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0% disk./.usageGB‚1260.8495407104492# disk.dm-0.out‚15646.88818359375 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚15646.88818359375 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0! proc.memory.rssMB‚ 2885.796875) disk./.usagePercent‚70.51330680275323 disk.drbd1817.out‚0 disk.sda.out‚ 117.296875 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd2041.in‚0( proc.memory.availableMB‚ 1940.3671875 network.recv‚292444 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 117.296875 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 network.sent‚ 4.230008e+06 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.742337544759113 cpu‚25.284322784104095 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 19.8212890625# memory_percent‚92.1046257019043Š1@Cž¸ ‚(:“ šŠÓž•Ç³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚282.715' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.23833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 í•êF¹ ‚(:»  šŠÓ’®è³ disk.drbd1817.in‚0 disk.drbd1289.in‚0 network.sent‚ 4.230074e+06% disk./.usageGB‚1260.8495788574219 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 19.8212890625 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0$ memory_percent‚92.10464159647623 disk.drbd2118.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1940.36328125 disk.sda.in‚0.03125# disk.dm-0.out‚15647.16943359375 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚15647.16943359375 cpu‚25.284208314773117* proc.memory.percent‚11.742337544759113 disk.drbd1965.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 network.recv‚292510) disk./.usagePercent‚70.51330893613164 disk.sda2.out‚ 117.71484375 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1356.in‚0! proc.memory.rssMB‚ 2885.796875 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 117.71484375 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0fö¹»“º ‚(:ˆ ÑšŠÓˆÂ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚277.377' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.459 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V"fC» ‚(:¸  ÑšŠÓÓ«ß³$ memory_percent‚92.10564295450845 disk.drbd2111.in‚0 network.sent‚ 4.235601e+06) disk./.usagePercent‚70.51321421413029 disk.dm-0.in‚ 62.01171875 disk.drbd1285.in‚0# disk.dm-0.out‚15647.39208984375 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 cpu‚25.28414936917522 disk.drbd2111.out‚0 disk.rbd1.out‚ 19.8837890625& disk.nvme0n1.out‚15647.39208984375 disk.drbd1356.in‚0$ disk./.usageGB‚1260.847885131836 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd1510.out‚0 disk.sda.out‚ 118.00390625 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0( proc.memory.availableMB‚ 1940.1171875 network.recv‚293586 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 118.00390625* proc.memory.percent‚11.742337544759113 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1226.out‚0 disk.drbd2041.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0ŒÐj®2¼ - _wandbruntime‚4370  _runtime‚4370¹í@£È½  ' lr-AdamW/pg1‚0.00011036942524822479 ' lr-AdamW/pg2‚0.00011036942524822479 trainer/global_step‚2399 % _timestamp‚1.7848436074822266e+09 _runtime‚ 4370.83946064 _step‚196¹ô Ô; Ç % _timestamp‚1.7848436074822266e+09 _runtime‚ 4370.83946064 _step‚196 ' lr-AdamW/pg1‚0.00011036942524822479 ' lr-AdamW/pg2‚0.00011036942524822479 trainer/global_step‚2399Äø§óä2¿ - _wandbruntime‚4370  _runtime‚4370™êOó'À íO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692181704670474 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚197 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20800664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3059143876608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚4370.843925581 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚689683033993840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20800664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692458630522368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚692458630522368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13693107 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7297263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7296920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚689683059421808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43061489 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22260825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3059094706688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848436074832783e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2502835396096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189190837607424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692457258912768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2502835396096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22260825 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192249981484032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692180334631220 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6395844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623zÖø'Á òO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692457258912768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2502835396096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22260825 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192249981484032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692180334631220 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6395844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692181704670474 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚197 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20800664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3059143876608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚4370.843925581 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚689683033993840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20800664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692458630522368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚692458630522368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13693107 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7297263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7296920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚689683059421808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43061489 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22260825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3059094706688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848436074832783e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2502835396096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189190837607424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755Å+]¢Ã2 - _wandbruntime‚4372  _runtime‚4372.‰1Ów à ÷M ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2498321439020 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2503857911296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43089022 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LD:àü†DeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22279985 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22269907 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20809037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚4372.318598864 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚2399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192329404643840 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚690234678243328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚690234678243328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43078938 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692738562328064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22269907 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚689961861195584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7300202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2502512467968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7299862 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43089022 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2502512467968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692461527246002 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3060404215808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692460157206636 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13698341 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43089022 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22279985 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22269907 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20809037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚4372.318598864 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚2399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192329404643840 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚690234678243328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚690234678243328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43078938 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692738562328064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22269907 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚689961861195584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7300202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2502512467968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7299862 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43089022 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2502512467968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692461527246002 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3060404215808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692460157206636 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13698341 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21219020 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚201 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22726841 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196241743730176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443331 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2554024952320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706455284603392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706453912993792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21219014 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2549749131474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚706172897242082 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 trainer/global_step‚2449 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22726841 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13970795 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703623122682640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706455284603392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21219020 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784843694690471e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21219014 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442956 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706171527202828 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703901233477632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193120155905024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 _runtime‚4458.051040285 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13971175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703901259651072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703901233477632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚706453912993792 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21219020 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚201 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22726841 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196241743730176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443331 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2554024952320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706455284603392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706453912993792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21219014 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2549749131474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚706172897242082 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 trainer/global_step‚2449 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22726841 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13970795 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703623122682640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706455284603392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21219020 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784843694690471e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21219014 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442956 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706171527202828 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703901233477632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193120155905024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 _runtime‚4458.051040285 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13971175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703901259651072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703901233477632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528É‚å>é2Ú - _wandbruntime‚4459  _runtime‚4459؃ Pü&Û öM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚372 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43953240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2553702024192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43963324 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6530484 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2555047467520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13976373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704180168941568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704180142768128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43953240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6530479 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22725859 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3122828748288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13976745 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193198374179840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚703901924456416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22725859 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22735937 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚704180142768128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚706452719817610 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706735216409088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706733844792320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2555047467520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22735937 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848436961678815e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196321141182464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2553702024192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21227381 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚ 4459.52551542 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7445894 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193198742270976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2549425321828 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3122767002624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2550769933226 trainer/global_step‚2449 _step‚202 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21227381 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚703901949884384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21227387 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚706735216409088ñ Ž¬"Ü ûM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13976745 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193198374179840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚703901924456416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22725859 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22735937 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚704180142768128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚706452719817610 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706735216409088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706733844792320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2555047467520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22735937 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 I@DeviceStatsMonitor.on_train_batch_end/segment.large½ ;ß_pool.current‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848436961678815e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196321141182464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2553702024192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21227381 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚ 4459.52551542 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7445894 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193198742270976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2549425321828 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3122767002624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2550769933226 trainer/global_step‚2449 _step‚202 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21227381 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚703901949884384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21227387 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚706735216409088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚372 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43953240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2553702024192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43963324 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6530484 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2555047467520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13976373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704180168941568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704180142768128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43953240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6530479 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22725859 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3122828748288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824Êy$Tó2Ý - _wandbruntime‚4459  _runtime‚4459ߧ–+Þ þ 0loss/SRqqt1/detection_loss‚1.176134100453438 .loss/FBt2/detection_loss‚0.644095145545828 &loss/total_loss‚11.446242486871203 trainer/global_step‚2449 _step‚203 2loss/SRqqt2/assignment_loss‚0.6186302776027024 2loss/FBt1/assignment_loss‚0.030128701920806758 /loss/FRt2/detection_loss‚1.5512543175890874 /loss/FRt1/assignment_loss‚2.274759324099503 2loss/SRqqt1/assignment_loss‚0.8200309515416034 0loss/FBt2/assignment_loss‚0.1647788006182412 /loss/FBt1/detection_loss‚1.8367835923643976 _runtime‚4459.561541065 1loss/SRqqt2/detection_loss‚0.6780893340089561 epoch‚8 $ _timestamp‚1.784843696204319e+09 0loss/FRt2/assignment_loss‚1.0558055989091069 /loss/FRt1/detection_loss‚0.5957523422175334õŽbš ß ƒ /loss/FRt1/detection_loss‚0.5957523422175334 0loss/SRqqt1/detection_loss‚1.176134100453438 .loss/FBt2/detection_loss‚0.644095145545828 &loss/total_loss‚11.446242486871203 trainer/global_step‚2449 _step‚203 2loss/SRqqt2/assignment_loss‚0.6186302776027024 2loss/FBt1/assignment_loss‚0.030128701920806758 /loss/FRt2/detection_loss‚1.5512543175890874 /loss/FRt1/assignment_loss‚2.274759324099503 2loss/SRqqt1/assignment_loss‚0.8200309515416034 0loss/FBt2/assignment_loss‚0.1647788006182412 /loss/FBt1/detection_loss‚1.8367835923643976 _runtime‚4459.561541065 1loss/SRqqt2/detection_loss‚0.6780893340089561 epoch‚8 $ _timestamp‚1.784843696204319e+09 0loss/FRt2/assignment_loss‚1.0558055989091069ˬB½O5à ‚(:ª  º›ŠÓÑ€ñ³ disk.drbd1550.in‚0 disk.drbd2118.out‚0" disk.dm-0.out‚15651.5927734375 disk.drbd2041.out‚0 disk.sda.out‚120.5 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1552.out‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1356.in‚0* proc.memory.percent‚11.742337544759113) disk./.usagePercent‚70.51329698921255 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 cpu‚25.283740175901308 disk.drbd1828.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 network.recv‚297719 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 20.2138671875 disk.drbd1584.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚15651.5927734375$ disk./.usageGB‚1260.849365234375 disk.sda2.out‚120.5 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 network.sent‚ 4.312621e+06 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0! proc.memory.rssMB‚ 2885.796875 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.10527737935385) proc.memory.availableMB‚ 1940.20703125Nã žá ‚(:“ º›ŠÓ²Éг gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚288.919' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.30633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KhΟâ ‚(:” É›ŠÓ°“æ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚309.989' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.32966666666665 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Å$ Cã ‚(:¸  É›ŠÓˆ­„´ disk.sda.in‚0.03125 disk.drbd1289.in‚0% disk.nvme0n1.out‚15651.8271484375 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0! proc.memory.rssMB‚ 2885.796875 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 cpu‚25.28379235737544 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda.out‚ 120.93359375 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0$ memory_percent‚92.10426012674967 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.sda2.out‚ 120.93359375) proc.memory.availableMB‚ 1940.45703125 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 20.2763671875 disk.nvme0n1.in‚ 62.01171875 network.sent‚ 4.312687e+06 network.recv‚297785 disk.dm-0.in‚ 62.01171875 disk.drbd1285.in‚0 disk.drbd1938.out‚0" disk.dm-0.out‚15651.8271484375 disk.drbd2078.out‚0% disk./.usageGB‚1260.8493461608887 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.742337544759113 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51329592252334 disk.sda1.out‚0ß+)Ÿä ‚(:” Ø›ŠÓå¢ã³ gpu.0.gpu‚83.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.869' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.62300000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çy•:å ‚(:¯  Ø›ŠÓÁÛ„´ disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 62.01171875# disk.nvme0n1.out‚15652.73828125 disk.drbd2118.in‚0 disk.drbd1289.in‚0 network.sent‚ 4.312753e+06$ disk./.usageGB‚1260.849365234375 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 20.2763671875 proc.cpu.threads‚17 network.recv‚297851 cpu‚25.283735655748895 disk.sda2.out‚ 121.265625 disk.sda.out‚ 121.265625 disk.drbd2078.out‚0) disk./.usagePercent‚70.51329698921255 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1940.19140625* proc.memory.percent‚11.742337544759113 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.drbd1285.in‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.sda2.in‚0.03125 disk.dm-0.out‚15652.73828125 disk.drbd2078.in‚0 disk.drbd1356.in‚0# memory_percent‚92.1053409576416 disk.drbd1931.in‚0HŽ¼iŸæ ‚(:” 益ÓæØdz gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.546' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.18199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÂ61ç ‚(:¦  益Ó©Øâ³ disk.nvme0n1.out‚ 15653.0625 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1828.out‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 network.recv‚299599 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2041.out‚0$ memory_percent‚92.10532506306967) disk./.usagePercent‚70.51329592252334 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.8493461608887 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 20.41015625 proc.cpu.threads‚17 network.sent‚ 4.37531e+06 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.dm-0.out‚ 15653.0625 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 62.01171875 cpu‚25.28357253377096 disk.sda.out‚ 121.6484375 disk.drbd1931.out‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 1940.1953125! proc.memory.rssMB‚ 2885.796875 disk.drbd1817.in‚0 disk.sda2.out‚ 121.6484375 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.742337544759113 disk.drbd2078.out‚0 disk.drbd1356.in‚0]‘þµè ‚(:’ ö›ŠÓ°Â·³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.02' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.33999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 <¼Ç?é ‚(:´  ö›ŠÓ‰”Ö³ disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.742337544759113 disk.drbd2118.in‚0 disk.drbd1817.in‚0 network.sent‚ 4.375376e+06 disk.drbd1552.in‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd1285.in‚0 disk.rbd1.in‚0$ memory_percent‚92.10432370503744) proc.memory.availableMB‚ 1940.44140625 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd2078.in‚0$ disk./.usageGB‚1260.849365234375 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚15653.427734375 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1938.out‚0! disk.dm-0.out‚15653.427734375 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0 network.recv‚299665 disk.nvme0n1.in‚ 62.01171875 disk.drbd1931.out‚0 disk.drbd1226.in‚0 cpu‚25.283572909080103 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 121.94921875 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51329698921255 disk.dm-0.in‚ 62.01171875 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.sda.out‚ 121.94921875 disk.drbd1938.in‚0 disk.rbd1.out‚ 20.41015625 disk.drbd1550.out‚0|ADÃ2ê - _wandbruntime‚4544  _runtime‚4544xQc±Èë  _runtime‚ 4544.83347168 _step‚204 ' lr-AdamW/pg1‚0.00010973046677773536 ' lr-AdamW/pg2‚0.00010973046677773536 trainer/global_step‚2499 % _timestamp‚1.7848437814762173e+09QÀæ®Íì Ç trainer/global_step‚2499 % _timestamp‚1.7848437814762173e+09 _runtime‚ 4544.83347168 _step‚204 ' lr-AdamW/pg1‚0.00010973046677773536 ' lr-AdamW/pg2‚0.00010973046677773536̇€.Ù2í - _wandbruntime‚4544  _runtime‚4544v Ôó'î íO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2603869072384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720450567074816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6659839 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚717846724175872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23182793 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚720164089813690 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21637364 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23172729 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720450567074816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2599508402996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200233016064512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197049106111488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2605214508544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2600853014282 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2605214508544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23182793 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚717846698002432 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44820163 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21637370 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23172729 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7590034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7848437814773319e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720451938684416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21637364 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197049474202624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚01S‰Â0ï òO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21637370 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23172729 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7590034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7848437814773319e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720451938684416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21637364 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197049474202624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2603869072384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720450567074816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6659839 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚717846724175872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23182793 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚720164089813690 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21637364 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23172729 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720450567074816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2599508402996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200233016064512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197049106111488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2605214508544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2600853014282 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2605214508544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23182793 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚717846698002432 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44820163ͶZPžð ‚(:“ …œŠÓóÞ¾³ gpu.0.gpu‚39.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚281.204' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.73466666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜¾Ð>ñ ‚(:³  …œŠÓçìß³ disk.drbd1673.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1931.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51329698921255 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 62.01171875$ disk./.usageGB‚1260.849365234375 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 122.32421875 disk.drbd1356.out‚0 disk.rbd1.out‚ 20.47265625 disk.drbd2118.in‚0 disk.drbd1552.in‚0 disk.drbd1828.out‚0$ disk.nvme0n1.out‚15653.626953125) proc.memory.availableMB‚ 1939.94140625 network.sent‚ 4.383579e+06 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.10635821024576 cpu‚25.28357124330773! proc.memory.rssMB‚ 2885.796875* proc.memory.percent‚11.742337544759113! disk.dm-0.out‚15653.626953125 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 122.32421875 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 network.recv‚300923 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0¿çÜÞ2ò - _wandbruntime‚4546  _runtime‚4546ˆ¢cvý&ó ÷M \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2606237023744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44827542 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21645737 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23191889 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718125633466368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23191889 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21645731 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6662484 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6662479 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720442542349852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720443912389218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717842038573184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4546.317006183 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14255795 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718125607292928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200313273771008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44827542 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21645737 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720731870490112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3185159845376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44837626 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200312852322816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23181811 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2606237023744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2604891580416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718125607292928 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2600529204636 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚2499 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3185213202432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848437829593592e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23181811 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197128060568576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197127692477440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14255425 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14255425 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6662479 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720442542349852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720443912389218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717842038573184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4546.317006183 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14255795 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718125607292928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200313273771008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44827542 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21645737 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720731870490112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3185159845376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44837626 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200312852322816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23181811 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2606237023744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2604891580416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718125607292928 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2600529204636 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚2499 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3185213202432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848437829593592e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23181811 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197128060568576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197127692477440ÎD&¶Ä2õ - _wandbruntime‚4546  _runtime‚4546elò§ö  2loss/SRqqt2/assignment_loss‚0.6163105513276984 /loss/FRt1/detection_loss‚0.6062840389913952 epoch‚8 1loss/SRqqt1/detection_loss‚1.2079466397561351 1loss/SRqqt2/detection_loss‚0.7505316689190046 /loss/FBt1/detection_loss‚1.5009471097588538 $ _timestamp‚1.784843782990665e+09 _runtime‚4546.347889029 /loss/FRt2/assignment_loss‚1.095670284703374 2loss/SRqqt1/assignment_loss‚0.8040111186050803 1loss/FBt2/assignment_loss‚0.12254409217185194 trainer/global_step‚2499 0loss/FRt1/assignment_loss‚2.1737055453701295 1loss/FBt1/assignment_loss‚0.05274064486700809 /loss/FRt2/detection_loss‚1.6339667726091907 0loss/FBt2/detection_loss‚0.48663221313609933 &loss/total_loss‚11.051290680215821 _step‚207?w0” ÷ † 0loss/FRt1/assignment_loss‚2.1737055453701295 1loss/FBt1/assignment_loss‚0.05274064486700809 /loss/FRt2/detection_loss‚1.6339667726091907 0loss/FBt2/detection_loss‚0.48663221313609933 &loss/total_loss‚11.051290680215821 _step‚207 2loss/SRqqt2/assignment_loss‚0.6163105513276984 /loss/FRt1/detection_loss‚0.6062840389913952 epoch‚8 1loss/SRqqt1/detection_loss‚1.2079466397561351 1loss/SRqqt2/detection_loss‚0.7505316689190046 /loss/FBt1/detection_loss‚1.5009471097588538 $ _timestamp‚1.784843782990665e+09 _runtime‚4546.347889029 /loss/FRt2/assignment_loss‚1.095670284703374 2loss/SRqqt1/assignment_loss‚0.8040111186050803 1loss/FBt2/assignment_loss‚0.12254409217185194 trainer/global_step‚2499ÏȆ­äžø ‚(:“ ”œŠÓä—׳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.811' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.27033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Úb‹9ù ‚(:®  ”œŠÓÉÓï³ disk.drbd1356.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.sda.out‚ 122.65625 disk.drbd1965.in‚0 disk.drbd1285.in‚0 cpu‚25.283405755500915 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1940.07421875 disk.sda2.in‚0.03125 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1226.out‚0! proc.memory.rssMB‚ 2885.796875 disk.dm-0.in‚ 62.01171875) disk./.usagePercent‚70.51329720255039$ disk.nvme0n1.out‚15654.044921875 disk.drbd2111.out‚0# memory_percent‚92.1058177947998* proc.memory.percent‚11.742337544759113% disk./.usageGB‚1260.8493690490723 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.sda2.out‚ 122.65625 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 20.47265625 network.recv‚301031 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0! disk.dm-0.out‚15654.044921875 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 network.sent‚ 4.383687e+06 disk.drbd1938.out‚0 disk.drbd1510.in‚0äk¯žú ‚(:“ £œŠÓ²¤·³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.663' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.88766666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞvŠû ‚(:¶  £œŠÓ£ªÖ³ disk.drbd1550.in‚0* proc.memory.percent‚11.742337544759113% disk./.usageGB‚1260.8493690490723 disk.drbd2078.in‚0# memory_percent‚92.1058177947998 disk.drbd1931.in‚0 disk.sda2.in‚0.03125 disk.drbd1226.in‚0 disk.sda2.out‚ 123.01953125! disk.dm-0.out‚15654.201171875 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 123.01953125 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.rbd1.in‚0 d·Ôæ+isk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 network.sent‚ 4.383753e+06 network.recv‚301097 disk.dm-0.in‚ 62.01171875 disk.rbd1.out‚ 20.6064453125 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0! proc.memory.rssMB‚ 2885.796875 cpu‚25.283462014030874 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 62.01171875$ disk.nvme0n1.out‚15654.201171875 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 1940.07421875) disk./.usagePercent‚70.51329720255039 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0À¾/•ü ‚(:’ ²œŠÓ™“³³ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚323.536' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚107.84533333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óc(z7ý ‚(:¬  ²œŠÓݟѳ disk.drbd1931.out‚0 disk.drbd1938.out‚0$ memory_percent‚92.10600852966309 disk.sda.out‚ 123.40625" disk.nvme0n1.out‚ 15654.5703125 disk.nvme0n1.in‚ 62.01171875 disk.sda2.out‚ 123.40625 disk.dm-0.out‚ 15654.5703125 disk.drbd2078.out‚0 network.recv‚302169 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0! proc.memory.rssMB‚ 2885.796875 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 network.sent‚ 4.389269e+06 disk.rbd1.out‚ 20.6064453125 disk.drbd1584.in‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 1940.02734375 disk.drbd1550.in‚0* proc.memory.percent‚11.742337544759113 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 62.01171875 cpu‚25.287841755513522 disk.drbd1226.in‚0 disk.drbd1356.in‚0$ disk./.usageGB‚1260.849349975586 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0) disk./.usagePercent‚70.51329613586118 disk.rbd1.in‚0 disk.drbd1828.out‚0 disk.drbd1673.in‚0hÖ#2þ - _wandbruntime‚4600  _runtime‚4600ÅZê•°;ÿ ªw 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5819712281227112 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7981473803520203 +particle/specificity‚0.9077380299568176 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862809181213379 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1805555671453476 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7332249879837036 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9948979616165161 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.648276150226593 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038982558995485306 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9917411208152771 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6643194556236267 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6664038300514221 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6961538791656494 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5845527052879333 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3770863115787506 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9821001291275024 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05447817221283913 *jet/accuracy_2_of_3‚0.5297448039054871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2190593034029007 % _timestamp‚1.7848438371884596e+09 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9884044528007507 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7128382921218872 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8392857313156128 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4692307710647583 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7971752882003784 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9683459401130676 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04655841365456581 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16059339046478271 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3832760155200958 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6420068740844727 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7411927580833435 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20636525750160217 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1930895894765854 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4549160897731781 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8142684698104858 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9832569360733032 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 )loss/val_total_loss‚10.38076025205977 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6834173798561096 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2075769007205963 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.207940474152565 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6840888261795044 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.37586408853530884 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.628677487373352 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16314278542995453 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901385307312012 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5297806262969971 particle/accuracy_1_of_2‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1805555671453476 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6071909666061401 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6171690225601196 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.6785714030265808 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21810415387153625 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8263888359069824 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8708156943321228 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1105116605758667 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6052743196487427 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2694854140281677 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6409462690353394 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6252513527870178 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20740221440792084 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19475522637367249 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6221856474876404 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992708683013916 +jet/accuracy_3_of_3‚0.05972786992788315 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5022607445716858 _step‚208 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7214184403419495 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1404111683368683 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08580339699983597 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9749338030815125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07387302070856094 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7920197248458862 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20088087022304535 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8472221493721008 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7083873748779297 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2727106213569641 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21159406006336212 trainer/global_step‚2519 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954710006713867 +jet/accuracy_2_of_2‚0.14170058071613312 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17068921029567719 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7957406044006348 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5302709341049194 *jet/accuracy_1_of_4‚0.9714088439941406 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5703322291374207 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.797301709651947 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023345643654465675 _runtime‚4600.546548373 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6352702379226685 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7916589379310608 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24531951546669006 *jet/accuracy_1_of_1‚0.3217860162258148 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07373573631048203 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 7validation_average_jet_accuracy‚0.36312511563301086 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2738926112651825 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950257539749146 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6914040446281433 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1266011893749237 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08573312312364578 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982993006706238 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16110120713710785 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9980158805847168 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6785714030265808 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18108287453651428 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7661917805671692 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9543830156326294 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6799601912498474 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.852996826171875 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1849973052740097 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 epoch‚8 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8872880935668945 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6850623488426208 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6489263772964478 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047718342393636703 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7450434565544128 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5452755689620972 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12412439286708832 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 +jet/accuracy_3_of_4‚0.09965380281209946 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23120546340942383 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43341442942619324 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9798952341079712 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7665926218032837 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30880722403526306 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22264111042022705 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8802432417869568 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4743589758872986 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22056365013122559 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2075769007205963 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6511648893356323 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8392857313156128 *jet/accuracy_1_of_2‚0.7767561674118042 )jet/accuracy_2_of_4‚0.812458872795105 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976958632469177 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 &particle/f_score‚0.635290801525116 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9597905278205872 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18058274686336517 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6227360963821411 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987577795982361 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6385250687599182 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3687892556190491 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8244996666908264 (particle/accuracy‚0.8465866446495056 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938791990280151 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5956043601036072 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5488888025283813 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6489263772964478 *"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04134411737322807 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9227613806724548 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.689767062664032 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3844945728778839 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4539306163787842 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7371231317520142 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8244996666908264 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9982993006706238 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07809150218963623 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05447817221283913 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632ŒÖ¯kõ7€¯w :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04134411737322807 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9227613806724548 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.689767062664032 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3844945728778839 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4539306163787842 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7371231317520142 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8244996666908264 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9982993006706238 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07809150218963623 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05447817221283913 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5819712281227112 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7981473803520203 +particle/specificity‚0.9077380299568176 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862809181213379 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904761910438538 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1805555671453476 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7332249879837036 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9948979616165161 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.648276150226593 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038982558995485306 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9917411208152771 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6643194556236267 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6664038300514221 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6961538791656494 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5845527052879333 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3770863115787506 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9821001291275024 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05447817221283913 *jet/accuracy_2_of_3‚0.5297448039054871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2190593034029007 % _timestamp‚1.7848438371884596e+09 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9884044528007507 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7128382921218872 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8392857313156128 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4692307710647583 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7971752882003784 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9683459401130676 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04655841365456581 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16059339046478271 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3832760155200958 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6420068740844727 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7411927580833435 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20636525750160217 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1930895894765854 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4549160897731781 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8142684698104858 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9832569360733032 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 )loss/val_total_loss‚10.38076025205977 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6834173798561096 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2075769007205963 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.207940474152565 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6840888261795044 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.37586408853530884 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.628677487373352 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16314278542995453 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901385307312012 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5297806262969971 particle/accuracy_1_of_2‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1805555671453476 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6071909666061401 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6171690225601196 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.6785714030265808 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21810415387153625 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8263888359069824 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8708156943321228 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1105116605758667 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6052743196487427 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2694854140281677 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6409462690353394 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6252513527870178 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20740221440792084 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19475522637367249 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6221856474876404 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992708683013916 +jet/accuracy_3_of_3‚0.05972786992788315 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5022607445716858 _step‚208 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7214184403419495 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1404111683368683 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08580339699983597 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9749338030815125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07387302070856094 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7920197248458862 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20088087022304535 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8472221493721008 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7083873748779297 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2727106213569641 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21159406006336212 trainer/global_step‚2519 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954710006713867 +jet/accuracy_2_of_2‚0.14170058071613312 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17068921029567719 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7957406044006348 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5302709341049194 *jet/accuracy_1_of_4‚0.9714088439941406 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5703322291374207 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.797301709651947 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023345643654465675 _runtime‚4600.546548373 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6352702379226685 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7916589379310608 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24531951546669006 *jet/accuracy_1_of_1‚0.3217860162258148 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07373573631048203 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 7validation_average_jet_accuracy‚0.36312511563301086 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2738926112651825 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950257539749146 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6914040446281433 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1266011893749237 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08573312312364578 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982993006706238 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16110120713710785 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9980158805847168 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6785714030265808 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18108287453651428 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7661917805671692 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9543830156326294 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6799601912498474 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.852996826171875 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1849973052740097 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 epoch‚8 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8872880935668945 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6850623488426208 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6489263772964478 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047718342393636703 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7450434565544128 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5452755689620972 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12412439286708832 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 +jet/accuracy_3_of_4‚0.09965380281209946 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23120546340942383 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43341442942619324 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9798952341079712 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7665926218032837 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30880722403526306 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22264111042022705 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8802432417869568 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4743589758872986 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22056365013122559 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2075769007205963 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6511648893356323 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8392857313156128 *jet/accuracy_1_of_2‚0.7767561674118042 )jet/accuracy_2_of_4‚0.812458872795105 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976958632469177 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBQ¶M¢Àt/event_purity‚0 particle/accuracy_3_of_4‚1 &particle/f_score‚0.635290801525116 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9597905278205872 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18058274686336517 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6227360963821411 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987577795982361 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6385250687599182 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3687892556190491 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8244996666908264 (particle/accuracy‚0.8465866446495056 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938791990280151 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5956043601036072 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5488888025283813 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6489263772964478 *"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5Ц¹cö‚(:’ ÁœŠÓŒçÔ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚57.0 gpu.0.powerWatts‚105.013' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.004333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î/¤¥=‚‚(:²  ÁœŠÓÞåô³ disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 cpu‚25.293321335327377 proc.cpu.threads‚17 network.sent‚ 4.454433e+06 disk.dm-0.in‚ 62.01171875 disk.drbd1285.out‚0* proc.memory.percent‚15.687386194864908 disk.drbd1510.in‚0 disk.drbd1931.out‚0$ disk./.usageGB‚1261.355125427246 disk.dm-0.out‚ 15926.7734375 disk.rbd1.in‚0 disk.rbd1.out‚ 20.7626953125$ memory_percent‚93.89189084370932 disk.drbd1817.in‚0 disk.sda2.out‚ 123.7578125 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.sda1.in‚0) disk./.usagePercent‚70.54158174682799 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 123.7578125 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1501.12890625# proc.memory.rssMB‚ 3855.33203125 disk.drbd2041.in‚0 disk.drbd2041.out‚0 network.recv‚304147 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0" disk.nvme0n1.out‚ 15926.7734375 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1226.out‚0 disk.drbd1552.out‚0âQÀ~9ƒ‚(:®  ÐœŠÓþ‚ô³ disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 20.9423828125 network.recv‚305290) disk./.usagePercent‚70.51329613586118 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.dm-0.out‚17594.21484375 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1673.out‚0# disk.nvme0n1.out‚17594.21484375 disk.drbd2118.in‚0 disk.sda.in‚0.03125$ memory_percent‚88.14926147460938 cpu‚25.290764312563823 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0% proc.memory.availableMB‚ 2912.4375$ disk./.usageGB‚1260.849349975586 disk.sda.out‚ 124.0390625! proc.memory.rssMB‚ 2885.953125 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 124.0390625 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.742973327636719 proc.cpu.threads‚17 network.sent‚ 4.457476e+06 disk.sda1.in‚0CoI¼š„‚(: ÐœŠÓæíÖ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚139.0' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚46.33333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬“…‚(:ˆ ßœŠÓÜþß³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚278.115' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XPGó8†‚(:­  ßœŠÓ¬…û³ network.recv‚305356) disk./.usagePercent‚70.51329613586118 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.sda.in‚0.03125 disk.sda2.out‚ 124.3046875 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2078.out‚0 network.sent‚ 4.457542e+06 disk.dm-0.in‚ 62.01171875 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0! disk.nvme0n1.out‚ 17594.359375 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0! proc.memory.rssMB‚ 2885.953125* proc.memory.percent‚11.742973327636719 disk.sda1.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0# memory_percent‚88.1492773691813 disk.drbd1938.in‚0 disk.rbd1.out‚ 21.1064453125 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0$ disk./.usageGB‚1260.849349975586 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.sda.out‚ 124.3046875 cpu‚25.290632693824758 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.dm-0.out‚ 17594.359375 disk.drbd2041.in‚0 disk.drbd2118.in‚0) proc.memory.availableMB‚ 2912.43359375ða+Z”‡‚(:‰ Óà¦á³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚305.835' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.945 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ûúô2ˆ‚(:§  ÓÎÕû³* proc.memory.percent‚11.742973327636719 disk.sda.out‚124.625 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0) proc.memory.availableMB‚ 2912.41796875 network.sent‚ 4.45765e+06 disk.dm-0.in‚ 62.01171875 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0$ memory_percent‚88.14934094746908 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 network.recv‚305464 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 21.2861328125 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1226.out‚0) disk./.usagePercent‚70.51329613586118 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.sda2.out‚124.625 disk.drbd1965.in‚0 disk.drbd1828.out‚0" disk.nvme0n1.out‚ 17595.0546875 cpu‚25.290662657772334$ disk./.usageGB‚1260.849349975586 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.dm-0.out‚ 17595.0546875†pR³2‰- _wandbruntime‚4656  _runtime‚4656%áOÈŠÂ ' lr-AdamW/pg1‚0.00010940090635643034 ' lr-AdamW/pg2‚0.00010940090635643034 trainer/global_step‚2549 $ _timestamp‚1.784843893147831e+09 _runtime‚4656.505080321 _step‚209<•Q¥Í‹Ç ' lr-AdamW/pg2‚0.00010940090635643034 trainer/global_step‚2549 $ _timestamp‚1.784843893147831e+09 _runtime‚4656.505080321 _step‚209 ' lr-AdamW/pg1‚0.00010940090635643034Ñéùl2Œ- _wandbruntime‚4656  _runtime‚4656ü »ó'íO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚322 _step‚210 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23686057 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚737014887929344 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7766015 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2549 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6798731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22145152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7765698 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚736718522336526 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45831209 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22145146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚737014887929344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201027792013312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚734352865425408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22145146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2660677067776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14564751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22145152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 410969600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3253814409216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848438931489341e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚737013516319744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204281649300992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚736719892375780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚734062301211408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204281238331392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚317 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2662022503936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2662022503936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3253857287680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2657565736404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6798736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2660677067776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚734352865425408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070ÚS ø'ŽòO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23675993 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚734352839251968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚734352839251968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 _runtime‚4656.506380797 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚322 _step‚210 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23686057 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚737014887929344 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7766015 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2549 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6798731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22145152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7765698 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚736718522336526 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45831209 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22145146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚737014887929344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201027792013312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚734352865425408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22145146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2660677067776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14564751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22145152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 410969600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3253814409216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848438931489341e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚737013516319744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204281649300992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚736719892375780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚734062301211408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204281238331392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚317 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2662022503936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2662022503936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3253857287680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2657565736404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6798736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2660677067776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚734352865425408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23695153 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚734341128413152 trainer/global_step‚2549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3255125895680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14570346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201106378379264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45848672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _runtime‚4657.983287121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3255083024384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2657241926758 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2661699575808 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45838588 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23685075 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚734631748542464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22153519 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23695153 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚734631774715904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚314 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737293448118272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204361093312512 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14570027 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45848672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22153519 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7768970 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398—êê'‘ûM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚734631748542464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22153519 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23695153 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚734631774715904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚314 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737293448118272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204361093312512 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14570027 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45848672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22153519 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7768970 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204361504274944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23695153 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚734341128413152 trainer/global_step‚2549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3255125895680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14570346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201106378379264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45848672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _runtime‚4657.983287121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3255083024384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2657241926758 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2661699575808 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45838588 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23685075 F2’- _wandbruntime‚4658  _runtime‚4658ÜñWá“‚ 0loss/FRt2/assignment_loss‚0.8715500906443917 2loss/SRqqt1/assignment_loss‚0.7988788389718021 1loss/FBt1/assignment_loss‚0.10541521959644992 /loss/FBt2/detection_loss‚0.6694445961731103 epoch‚9 0loss/FRt1/assignment_loss‚2.1305829653193507 &loss/total_loss‚10.423656754058936 /loss/FRt1/detection_loss‚0.6101460030328801 /loss/FRt2/detection_loss‚1.5734544805159183 /loss/FBt1/detection_loss‚1.2313777948064464 trainer/global_step‚2549 % _timestamp‚1.7848438946617415e+09 _step‚212 2loss/SRqqt2/assignment_loss‚0.5630392596626539 1loss/FBt2/assignment_loss‚0.06310662960179506 1loss/SRqqt1/detection_loss‚1.1132118456464792 1loss/SRqqt2/detection_loss‚0.6934490300876583 _runtime‚4658.018963176@bh ”‡ /loss/FRt1/detection_loss‚0.6101460030328801 /loss/FRt2/detection_loss‚1.5734544805159183 /loss/FBt1/detection_loss‚1.2313777948064464 trainer/global_step‚2549 % _timestamp‚1.7848438946617415e+09 _step‚212 2loss/SRqqt2/assignment_loss‚0.5630392596626539 1loss/FBt2/assignment_loss‚0.06310662960179506 1loss/SRqqt1/detection_loss‚1.1132118456464792 1loss/SRqqt2/detection_loss‚0.6934490300876583 _runtime‚4658.018963176 0loss/FRt2/assignment_loss‚0.8715500906443917 2loss/SRqqt1/assignment_loss‚0.7988788389718021 1loss/FBt1/assignment_loss‚0.10541521959644992 /loss/FBt2/detection_loss‚0.6694445961731103 epoch‚9 0loss/FRt1/assignment_loss‚2.1305829653193507 &loss/total_loss‚10.423656754058936Ôž¼ Ÿ•‚(:” ýœŠÓ™¡Ï³ gpu.0.gpu‚29.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚109.432' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.477333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ê7 ?–‚(:´  ýœŠÓ›äí³ disk.drbd1584.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51329613586118 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1931.in‚0 disk.drbd1938.out‚0$ disk./.usageGB‚1260.849349975586$ memory_percent‚88.14956347147623 cpu‚25.290587264051922 network.sent‚ 4.511747e+06 network.recv‚307126 disk.sda2.in‚0.03125 disk.drbd2078.out‚0) proc.memory.availableMB‚ 2912.36328125 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1673.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 125.01953125 disk.drbd1289.out‚0 disk.sda1.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 125.01953125 disk.rbd1.out‚ 21.3486328125# disk.nvme0n1.out‚17595.27734375 disk.sda1.in‚0 disk.dm-0.out‚17595.27734375 disk.drbd1965.in‚0* proc.memory.percent‚11.742973327636719 disk.drbd1356.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 62.01171875 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚07V>‰ž—‚(:“ ŒŠÓ’€±³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.627' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.54233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pß,71˜‚(:¦  ŒŠÓÀ§Ñ³ disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.sda2.out‚125.3125 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1550.in‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0! proc.memory.rssMB‚ 2885.953125$ memory_percent‚88.14957936604819 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 21.3486328125 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 62.01171875 disk.drbd2041.out‚0 disk.drbd1828.in‚0 cpu‚25.29072585023691 network.sent‚ 4.511813e+06 disk.drbd1226.out‚0 disk.drbd2118.in‚0! disk.nvme0n1.out‚ 17595.734375' proc.memory.availableMB‚ 2912.359375* proc.memory.percent‚11.742973327636719 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.sda.out‚125.3125 disk.dm-0.out‚ 17595.734375 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 network.recv‚307192) disk./.usagePercent‚70.51329720255039% disk./.usageGB‚1260.8493690490723xÒ_ž™‚(:“ ›ŠÓʆ¶³ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.066' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.35533333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷Š –@š‚(:µ  ›ŠÓ¤×Ù³ disk.dm-0.in‚ 62.01171875 disk.drbd1828.out‚0& disk.nvme0n1.out‚17596.40771484375 disk.drbd1510.out‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 21.482421875 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 125.5703125 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0$ memory_percent‚88.14957936604819 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0# disk.dm-0.out‚17596.40771484375 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.742973327636719) disk./.usagePercent‚70.51306573099306 disk.drbd2118.in‚0 disk.drbd1550.out‚0$ disk./.usageGB‚1260.845230102539 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 125.5703125 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2885.953125 network.recv‚307300 disk.drbd1817.out‚0 disk.drbd1938.out‚0 cpu‚25.290860198796246 network.sent‚ 4.511921e+06 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0' proc.memory.availableMB‚ 2912.359375 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1931.in‚0 rs€ž›‚(:“ ªŠÓ´ÚÖ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚285.241' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.08033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rUõÓAœ‚(:¶  ªŠÓž»÷³ disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0$ memory_percent‚88.14961115519205 network.sent‚ 4.577029e+06 disk.drbd1965.in‚0 disk.drbd2111.out‚0( proc.memory.availableMB‚ 2912.3515625 proc.cpu.threads‚17 disk.drbd1938.out‚0) disk./.usagePercent‚70.51306679768227 disk.nvme0n1.in‚ 62.01171875 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0# disk.dm-0.out‚17596.81298828125 disk.rbd1.out‚ 21.482421875& disk.nvme0n1.out‚17596.81298828125 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.8452491760254 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 125.9765625 disk.sda2.out‚ 125.9765625 disk.drbd1828.out‚0* proc.memory.percent‚11.742973327636719 disk.drbd1931.out‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 network.recv‚308922 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 cpu‚25.29072841117129 disk.drbd1828.in‚0¯‚h¢“‚(:ˆ ¹ŠÓ¹Í®³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚173.361' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚57.787 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q.t=ž‚(:²  ¹ŠÓ¿®Í³ disk.sda1.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51306402429034 disk.drbd1828.out‚0 disk.rbd1.out‚ 21.544921875 disk.drbd1356.out‚0$ disk./.usageGB‚1260.845199584961 disk.drbd1226.out‚0 disk.drbd1289.out‚0# disk.dm-0.out‚17597.03173828125 disk.drbd2078.out‚0$ memory_percent‚88.14961115519205 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 cpu‚25.290656219881274 disk.nvme0n1.in‚ 62.01171875 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 126.40625 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚17597.03173828125 disk.sda2.in‚0.03125 disk.sda.in‚0.03125 disk.drbd1673.out‚0! proc.memory.rssMB‚ 2885.953125 network.sent‚ 4.577137e+06 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 2912.3515625* proc.memory.percent‚11.742973327636719 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 62.01171875 disk.drbd1285.out‚0 network.recv‚309030 disk.drbd2111.out‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 126.40625 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0VòºŸ‚(:’ Ȋӷܺ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚295.09' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.36333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓËg? ‚(:´  ÈŠÓþûÕ³ disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 126.74609375 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 2912.3359375 disk.sda1.in‚0 disk.rbd1.out‚ 21.544921875 network.sent‚ 4.585336e+06 disk.dm-0.in‚ 62.01171875 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0( disk./.usagePercent‚70.5130706377634 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0! disk.dm-0.out‚17598.029296875 disk.drbd1550.out‚0 disk.drbd2111.out‚0! proc.memory.rssMB‚ 2885.953125 network.recv‚310224 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 62.01171875 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0$ memory_percent‚88.14967473347983 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0% disk./.usageGB‚1260.8453178405762 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.742973327636719$ disk.nvme0n1.out‚17598.029296875 disk.drbd1584.in‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 cpu‚25.290632467561263 disk.drbd1552.in‚0 disk.sda.out‚ 126.74609375 disk.drbd2041.out‚0›;p32¡- _wandbruntime‚4743  _runtime‚4743%Ãj•È¢Â ' lr-AdamW/pg1‚0.00010906468244848177 ' lr-AdamW/pg2‚0.00010906468244848177 trainer/global_step‚2599 % _timestamp‚1.7848439801385813e+09 _runtime‚ 4743.49689884 _step‚213n4vꣂ trainer/global_step‚2599 % _timestamp‚1.7848439801385813e+09 _runtime‚ 4743.49689884 _step‚213 ' lr-AdamW/pg1‚0.00010906468244848177 ' lr-AdamW/pg2‚0.00010906468244848177Õ¢Ûì2¤- _wandbruntime‚4743  _runtime‚4743‰ò戥ìO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748002389900208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚751010170400768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚748298329950208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24131945 QIDeviceStatsMonitor.on_train_batch_start/x#/DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22563496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204956742219776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2711866624000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7913148 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚750709714908134 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46695441 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3316440787968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2713212060160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2713212060160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6930731 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7912802 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2708669619212 trainer/global_step‚2599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚751011542010368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 4743.49701521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22563496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚751011542010368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46705511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22563502 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚750711084947388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚748298329950208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24142009 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14843533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864VD6ª÷'¦ñO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22563496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204956742219776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2711866624000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7913148 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚750709714908134 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46695441 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3316440787968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2713212060160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2713212060160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6930731 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7912802 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2708669619212 trainer/global_step‚2599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚751011542010368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 4743.49701521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22563496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚751011542010368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46705511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22563502 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚750711084947388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚748298329950208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24142009 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14843533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748002389900208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚751010170400768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚748298329950208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24131945 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2707325007926 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2711866624000 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24131945 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22563502 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚751010170400768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚748298303776768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24142009 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755ÖCæbÙ2§- _wandbruntime‚4744  _runtime‚4744]jôý&¨÷M K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14849095 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208352968292352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205035328585728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6933376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚748281191673984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14849450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46712890 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚748577239240704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2714234575360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24151105 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2712889132032 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46722974 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚751291473816064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205035696676864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3317639706624 _runtime‚4744.976568083 _step‚215 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22571863 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚750990907522916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2714234575360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚748577239240704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚748577213067264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2709690420964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24141027 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2708345809566 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22571869 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚751290102199296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7915724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24141027 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355†ÞG‡M ©üM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚748577213067264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22571869 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7916074 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚208353393934848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46722974 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 trainer/global_step‚2599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6933371 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46712890 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46722974 RCDeviceStatsMonitor.on_train_batch_en³\É`µd/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚751291473816064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205035696676864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3317639706624 _runtime‚4744.976568083 _step‚215 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22571863 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚750990907522916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2714234575360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚748577239240704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚748577213067264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2709690420964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24141027 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2708345809566 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22571869 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚751290102199296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7915724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24141027 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14849095 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208352968292352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205035328585728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6933376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚748281191673984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14849450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46712890 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚748577239240704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2714234575360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24151105 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2712889132032 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚765006824481792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2763056180224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15122647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚764700907479742 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚762243768301568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7062736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚218 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762243768301568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208886060517376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47579813 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24587897 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3378933005824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24597961 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761942504016976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47569743 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7062731 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚761942478589008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981852 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212264944353280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 $ _timestamp‚1.784844067159411e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚365 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚370 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208886428608512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚26491þT4{¾ñO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚765006824481792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3378883835904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚765006824481792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2763056180224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15122647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚764700907479742 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚762243768301568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7062736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚218 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762243768301568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208886060517376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47579813 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24587897 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3378933005824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24597961 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761942504016976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47569743 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7062731 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚761942478589008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981852 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212264944353280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 $ _timestamp‚1.784844067159411e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚365 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚370 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208886428608512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112ÚR<ºJ2¿- _wandbruntime‚4831  _runtime‚4831Á—û~ý&À÷M UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2764078688256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762522677592064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47597276 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208964646883328 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762221280362784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚4831.997580817 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47587192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2765424131584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚373 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2764078688256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24596979 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2765424131584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212344785000960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚765288127897088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47587192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7065376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚762522703765504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762221305790752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24607057 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3380138117632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15127831 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚765286756280320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24596979 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8062828 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47597276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22990213 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 % _timestamp‚1.7848440686399655e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3380193571840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15128204 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22990213 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7065371 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¬”‰&'ÁüM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚764980730055158 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208964646883328 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762221280362784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚4831.997580817 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47587192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2765424131584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚373 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2764078688256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24596979 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2765424131584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212344785000960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚765288127897088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47587192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7065376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚762522703765504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762221305790752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24607057 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3380138117632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15127831 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚765286756280320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24596979 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8062828 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47597276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22990213 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 % _timestamp‚1.7848440686399655e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3380193571840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15128204 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22990213 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7065371 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2764078688256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762522677592064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47597276 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25043849 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441381166592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15402192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7848441542057886e+09 _step‚222 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779003478562816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25043849 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2815591172608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15401826 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8207456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2815591172608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚778692100051350 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚775882592705776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7194736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864^Ì}+ø'ÔòO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3441436627968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚779003478562816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48444045 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23400196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚778693470090604 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25043849 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441381166592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15402192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7848441542057886e+09 _step‚222 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779003478562816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25043849 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2815591172608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15401826 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8207456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2815591172608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚778692100051350 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚775882592705776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7194736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864Þ/ÿæ˜2Õ- _wandbruntime‚4919  _runtime‚4919Ê'AÖöM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2815268244480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212893965180928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216337014102016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48471578 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25063009 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚778973292666132 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚776161394479552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23408569 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚776468142116864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2816613687808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48461494 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15407399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2811898186580 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25052931 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚776468142116864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚778971922626766 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_bÔ•Ú áatch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25052931 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8210384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _step‚223 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2810553575182 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779284781978112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚779284781978112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23408563 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779283410361344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚776161369051584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2816613687808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7197376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8210028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005òG96'×ûM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216337014102016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48471578 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25063009 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚778973292666132 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚776161394479552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23408569 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚776468142116864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2816613687808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48461494 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15407399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2811898186580 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25052931 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚776468142116864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚778971922626766 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25052931 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8210384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _step‚223 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2810553575182 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779284781978112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚779284781978112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23408563 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779283410361344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚776161369051584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2816613687808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7197376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8210028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2815268244480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212893965180928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366ß­ ªæ2Ø- _wandbruntime‚4919  _runtime‚4919?C«Ùý 0loss/FRt1/assignment_loss‚2.0876886883596093 0loss/FRt2/assignment_loss‚0.8783724370605905 2loss/SRqqt2/assignment_loss‚0.5989080715467039 /loss/FRt1/detection_loss‚0.5689098520856635 /loss/FRt2/detection_loss‚1.6680241619505718 1loss/SRqqt1/detection_loss‚0.9171274105777903 0loss/SRqqt2/detection_loss‚0.701056624136681 .loss/FBt1/detection_loss‚1.163887069944073 -loss/FBt2/detection_loss‚0.66615865613841 _runtime‚4919.076872574 %loss/total_loss‚10.13020947228385 trainer/global_step‚2699 % _timestamp‚1.7848441557196634e+09 _step‚224 2loss/SRqqt1/assignment_loss‚0.7562071843380709 1loss/FBt1/assignment_loss‚0.06966675547193839 1loss/FBt2/assignment_loss‚0.05420256067374714 epoch‚9yÂáæÚ‚ 2loss/SRqqt1/assignment_loss‚0.7562071843380709 1loss/FBt1/assignment_loss‚0.06966675547193839 1loss/FBt2/assignment_loss‚0.05420256067374714 epoch‚9 0loss/FRt1/assignment_loss‚2.0876886883596093 0loss/FRt2/assignment_loss‚0.8783724370605905 2loss/SRqqt2/assignment_loss‚0.5989080715467039 /loss/FRt1/detection_loss‚0.5689098520856635 /loss/FRt2/detection_loss‚1.6680241619505718 1loss/SRqqt1/detection_loss‚0.9171274105777903 0loss/SRqqt2/detection_loss‚0.701056624136681 .loss/FBt1/detection_loss‚1.163887069944073 -loss/FBt2/detection_loss‚0.66615865613841 _runtime‚4919.076872574 %loss/total_loss‚10.13020947228385 trainer/global_step‚2699 % _timestamp‚1.7848441557196634e+09 _step‚224àpê¡LBÛ‚(:·  üžŠÓØšî³ disk.nvme0n1.in‚ 62.1015625 disk.drbd2118.in‚0 disk.sda.in‚0.03125 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚17602.97119140625 disk.drbd1817.in‚0 disk.sda2.out‚ 130.94140625 cpu‚25.28996145530157 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.742973327636719 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0# memory_percent‚92.1042283376058 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 62.1015625 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 22.3994140625 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 network.sent‚ 4.725207e+06 network.recv‚316769% disk./.usageGB‚1260.8465728759766 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0# disk.dm-0.out‚17602.97119140625 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1940.46484375 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 130.94140625 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0) disk./.usagePercent‚70.51314082591304 disk.drbd1552.out‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1356.in‚0'"2žÜ‚(:“ üžŠÓƒÐѳ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚296.525' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.84166666666665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g‚´œ“Ý‚(:ˆ ‹ŸŠÓ®ªî³ gpu.0.gpu‚53.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.867' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.289 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žï¤CÞ‚(:¸  ‹ŸŠÓÄðƒ´ disk.drbd2118.in‚0 disk.dm-0.in‚ 62.1015625 disk.drbd1965.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1289.in‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1940.21484375 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚17603.20166015625 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0# disk.dm-0.out‚17603.20166015625 disk.drbd2041.in‚0 disk.rbd1.out‚ 22.5634765625 disk.nvme0n1.in‚ 62.1015625 network.sent‚ 4.800594e+06 disk.drbd1510.in‚0 disk.drbd1550.out‚0 cpu‚25.28993965483622 network.recv‚318628% disk./.usageGB‚1260.8465728759766 disk.drbd1584.in‚0 disk.drbd1552.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.742973327636719) disk./.usagePercent‚70.51314082591304 disk.sda.out‚ 131.22265625 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0$ memory_percent‚92.10524559020996 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.sda2.out‚ 131.22265625 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚04’Ÿß‚(:” šŸŠÓñ¹Ò³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚309.496' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.16533333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÅ,¿<à‚(:±  šŸŠÓéŒì³ disk.drbd2041.in‚0) proc.memory.availableMB‚ 1940.21484375% disk./.usageGB‚1260.8465728759766 disk.drbd2078.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 network.sent‚ 4.80066e+06 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0* proc.memory.percent‚11.742973327636719 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0# disk.dm-0.out‚17603.53564453125! proc.memory.rssMB‚ 2885.953125 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd1817.out‚0 disk.drbd1356.in‚0 cpu‚25.28992171397775 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 22.7431640625 disk.drbd2118.in‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.51314082591304 disk.drbd1931.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚17603.53564453125 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 62.1015625 disk.sda2.out‚ 131.71875 network.recv‚318694 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125$ memory_percent‚92.10524559020996 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 131.71875 disk.drbd1289.out‚0ÒÒŒžá‚(:“ ©ŸŠÓ‚µ³ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.755' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.58500000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú:ÿ ?â‚(:´  ©ŸŠÓê­×³ disk.nvme0n1.in‚ 62.1015625 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 1940.13671875* proc.memory.percent‚11.742973327636719 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.sda.in‚0.03125# disk.dm-0.out‚17603.73095703125 cpu‚25.28990366135621 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0$ memory_percent‚92.10556348164877 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 132.015625 disk.drbd1289.out‚0 network.sent‚ 4.863245e+06 disk.rbd1.out‚ 22.8056640625 disk.sda1.in‚0 disk.sda2.out‚ 132.015625 disk.dm-0.in‚ 62.1015625) disk./.usagePercent‚70.51313975922385& disk.nvme0n1.out‚17603.73095703125 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 network.recv‚320488% disk./.usageGB‚1260.8465538024902 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0J–I±žã‚(:“ ¸ŸŠÓªØ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚280.795' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.59833333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡’]Aä‚(:¶  ¸ŸŠÓï˜å³% disk./.usageGB‚1260.8465538024902 disk.dm-0.in‚ 62.1015625* proc.memory.percent‚11.742973327636719 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51313975922385 disk.drbd1938.in‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1584.in‚0 network.sent‚ 4.863353e+06 disk.sda.out‚ 132.28515625 disk.drbd1931.in‚0 disk.rbd1.out‚ 22.8056640625 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0$ memory_percent‚92.10548400878906 disk.drbd2118.out‚0 disk.drbd1510.in‚0! proc.memory.rssMB‚ 2885.953125 network.recv‚320596 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0& proc.memory.availableMB‚ 1940.15625 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 62.1015625& disk.nvme0n1.out‚17604.05224609375 disk.sda2.out‚ 132.28515625 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚17604.05224609375 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 cpu‚25.289733295641152 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0­Yéf”å‚(:‰ ÇŸŠÓáõ¶³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.746' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•G1µ@æ‚(:µ  ÇŸŠÓ½éÓ³ disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 cpu‚25.289715639277826& disk.nvme0n1.out‚17604.20068359375 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 132.62109375& proc.memory.availableMB‚ 1940.15625 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.10548400878906 disk.sda1.in‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.742973327636719 network.sent‚ 4.863353e+06 disk.sda.in‚0.03125 disk.drbd1510.out‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1673.in‚0# disk.dm-0.out‚17604.20068359375 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 network.recv‚320596) disk./.usagePercent‚70.51313975922385 disk.drbd2118.out‚0 disk.rbd1.out‚ 22.939453125 disk.drbd1285.out‚0% disk./.usageGB‚1260.8465538024902 disk.drbd1817.out‚0 disk.dm-0.in‚ 62.1015625 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.sda.out‚ 132.62109375 disk.drbd1931.out‚0 disk.drbd1550.in‚0¿ùÂã2ç- _wandbruntime‚5004  _runtime‚5004döœªÉèà ' lr-AdamW/pg1‚0.00010801647484966323 ' lr-AdamW/pg2‚0.00010801647484966323 trainer/global_step‚2749 % _timestamp‚1.7848442412809412e+09 _runtime‚5004.638201914 _step‚225)¨ÚÎéÈ % _timestamp‚1.7848442412809412e+09 _runtime‚5004.638201914 _step‚225 ' lr-AdamW/pg1‚0.00010801647484966323 ' lr-AdamW/pg2‚0.00010801647484966323 trainer/global_step‚2749á= Ž2ê- _wandbruntime‚5004  _runtime‚5004Rx!c+ ëíO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚789822655966608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23818552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚790134697351168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚790134723524608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 431941120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49328417 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8354409 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866780728832 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7326731 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚391 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚216744697112576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792683292622958 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23818546 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25499801 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793000132643840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7326736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚793001504253440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25509865 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2861981267636 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865435292672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚226 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚63849984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23818552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3503886653952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49318347 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23818546 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220248951857664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220248519916544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789822681394576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15681536 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚2749 % _timestamp‚1.7848442412820954e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25509865 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25499801 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚216745065203712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865435292672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚5004.642950115 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793001504253440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866780728832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚790134723524608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚790134697351168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚396(*ø'ìòO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25509865 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2861981267636 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865435292672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚226 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚63849984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23818552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3503886653952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49318347 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23818546 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220248951857664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220248519916544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789822681394576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15681536 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚2749 % _timestamp‚1.7848442412820954e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25509865 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25499801 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚216745065203712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865435292672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚5004.642950115 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793001504253440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866780728832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚790134723524608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚790134697351168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚396 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚789822655966608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23818552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚790134697351168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚790134723524608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 431941120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49328417 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8354409 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866780728832 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7326731 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚391 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚216744697112576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792683292622958 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23818546 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25499801 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793000132643840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7326736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚793001504253440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672âfÛ$£2í- _wandbruntime‚5006  _runtime‚5006ÏŠ+•ý&î÷M OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚378 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8357325 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216823283478528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25508883 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216823651569664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚792964485237740 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚790413632815104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23826913 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793281436059136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7329371 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2866457800704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚792963115198374 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49335796 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867803244032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚383 _step‚227 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8357703 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚793281436059136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚790101457740384 % _timestamp‚1.7848442427643633e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49345880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚5006.121991106 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15686696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49345880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23826919 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3505119572480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25518961 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7329376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867803244032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3505062021120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15687079 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2863002069388 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49335796 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚790413606641664e~ŠvîïüM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚5006.121991106 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15686696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49345880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23826919 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3505119572480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25518961 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7329376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867803244032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3505062021120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15687079 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2863002069388 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.larg ÍõŸe_pool.current‚ 47515172864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49335796 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚790413606641664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚378 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8357325 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216823283478528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25508883 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216823651569664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚792964485237740 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚790413632815104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23826913 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793281436059136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7329371 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2866457800704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚792963115198374 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49335796 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867803244032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚383 _step‚227 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8357703 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚793281436059136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚790101457740384 % _timestamp‚1.7848442427643633e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49345880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360ã™ó×E2ð- _wandbruntime‚5006  _runtime‚5006H\Ôñ‚ /loss/FRt1/detection_loss‚0.5715252050430606 /loss/FRt2/detection_loss‚1.6807978482350725 &loss/total_loss‚10.509386333083757 % _timestamp‚1.7848442427993758e+09 /loss/FRt1/assignment_loss‚2.146447632003289 2loss/SRqqt1/assignment_loss‚0.6008149332810204 2loss/FBt1/assignment_loss‚0.006670148428767675 /loss/FBt2/detection_loss‚0.6217257329690229 trainer/global_step‚2749 _step‚228 2loss/SRqqt2/assignment_loss‚0.5901519299994492 /loss/FBt1/detection_loss‚1.4191626828163861 epoch‚9 0loss/FRt2/assignment_loss‚1.0515384300493857 1loss/FBt2/assignment_loss‚0.09722549021331615 1loss/SRqqt1/detection_loss‚1.0577180467309877 1loss/SRqqt2/detection_loss‚0.6656082533139994 _runtime‚5006.156582148Ã3 ¬ ò‡ 2loss/SRqqt1/assignment_loss‚0.6008149332810204 2loss/FBt1/assignment_loss‚0.006670148428767675 /loss/FBt2/detection_loss‚0.6217257329690229 trainer/global_step‚2749 _step‚228 2loss/SRqqt2/assignment_loss‚0.5901519299994492 /loss/FBt1/detection_loss‚1.4191626828163861 epoch‚9 0loss/FRt2/assignment_loss‚1.0515384300493857 1loss/FBt2/assignment_loss‚0.09722549021331615 1loss/SRqqt1/detection_loss‚1.0577180467309877 1loss/SRqqt2/detection_loss‚0.6656082533139994 _runtime‚5006.156582148 /loss/FRt1/detection_loss‚0.5715252050430606 /loss/FRt2/detection_loss‚1.6807978482350725 &loss/total_loss‚10.509386333083757 % _timestamp‚1.7848442427993758e+09 /loss/FRt1/assignment_loss‚2.146447632003289äÙ"ÿŸó‚(:” ÖŸŠÓғƳ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚325.162' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚108.38733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NKgAô‚(:¶  ÖŸŠÓÇšæ³ disk.dm-0.in‚ 62.1015625 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 62.1015625* proc.memory.percent‚11.742973327636719) disk./.usagePercent‚70.51316855983235$ disk./.usageGB‚1260.847068786621 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚17605.41845703125 disk.rbd1.out‚ 22.939453125 disk.drbd1550.in‚0 disk.drbd1584.out‚0& disk.nvme0n1.out‚17605.41845703125 network.recv‚321671 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 cpu‚25.289794281542935 disk.sda2.in‚0.03125 disk.sda2.out‚ 132.85546875 disk.drbd1356.out‚0 disk.rbd1.in‚0# memory_percent‚92.1058177947998) proc.memory.availableMB‚ 1940.07421875 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 network.sent‚ 4.868945e+06 disk.drbd1938.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 132.85546875 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0méaÒžõ‚(:“ 埊ÓùÑܳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.486' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.82866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pxu?ö‚(:´  埊Ó—Àû³ disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.742973327636719 network.sent‚ 4.869053e+06 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1817.out‚0! proc.memory.rssMB‚ 2885.953125 network.recv‚321779 disk.drbd1285.in‚0 disk.sda.out‚ 133.37890625 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.sda1.out‚0 disk.drbd2118.out‚0# disk./.usageGB‚1260.84712600708 disk.drbd2078.out‚0 disk.drbd1226.out‚0# memory_percent‚92.1058177947998 proc.cpu.threads‚17 disk.nvme0n1.in‚ 62.1015625 disk.sda2.out‚ 133.37890625# disk.dm-0.out‚17605.73486328125 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 cpu‚25.28967452719018 disk.dm-0.in‚ 62.1015625 disk.drbd1356.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚17605.73486328125 disk.drbd1817.in‚0 disk.drbd1673.out‚0) proc.memory.availableMB‚ 1940.07421875 disk.drbd1356.in‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51317175989998 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 23.001953125 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0ÓpÕ|Ÿ÷‚(:” ôŸŠÓõ—ë³ gpu.0.gpu‚70.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.567' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.85566666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âh!8ø‚(:­  ôŸŠÓÎËŠ´ disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 network.recv‚321845 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 1940.07421875 cpu‚25.2896052634783 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.sda2.out‚133.6875# memory_percent‚92.1058177947998 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚17607.17626953125) disk./.usagePercent‚70.51315191948076 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 62.1015625& disk.nvme0n1.out‚17607.17626953125 disk.drbd1817.in‚0 disk.sda.out‚133.6875 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 network.sent‚ 4.869119e+06 disk.drbd1510.in‚0! proc.memory.rssMB‚ 2885.953125* proc.memory.percent‚11.742973327636719 disk.drbd2078.out‚0 disk.rbd1.out‚ 23.001953125% disk./.usageGB‚1260.8467712402344 disk.sda.in‚0.03125 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0é‘M~žù‚(:“ ƒ ŠÓïšÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚305.53' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.84333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É(™@ú‚(:µ  ƒ ŠÓù³ disk.drbd2111.in‚0& disk.nvme0n1.out‚17607.48486328125 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 133.9765625 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 23.1357421875 disk.drbd1817.in‚0 disk.drbd1289.in‚0$ memory_percent‚92.10583368937174 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125( proc.memory.availableMB‚ 1940.0703125 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 133.9765625 disk.drbd1938.in‚0 disk.drbd1356.in‚0 cpu‚25.28948992858243) disk./.usagePercent‚70.51317431995405 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0# disk.dm-0.out‚17607.48486328125 disk.drbd1965.out‚0 network.sent‚ 4.877326e+06 network.recv‚323064 disk.drbd2111.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.742973327636719 disk.drbd1285.out‚0! proc.memory.rssMB‚ 2885.953125 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd1550.out‚0 disk.dm-0.in‚ 62.1015625% disk./.usageGB‚1260.8471717834473 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2041.in‚0ú Ü'žû‚(:“ ’ ŠÓ—³Ñ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚269.384' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.79466666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 £ŽšCü‚(:¸  ’ ŠÓŠœó³ disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125# disk.dm-0.out‚17608.60107421875) proc.memory.availableMB‚ 1940.07421875! proc.memory.rssMB‚ 2885.953125 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 62.1015625 disk.dm-0.in‚ 62.1015625 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0# memory_percent‚92.1058177947998 cpu‚25.289423341843868 disk.drbd1226.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.742973327636719 disk.rbd1.out‚ 23.1357421875 disk.drbd1817.in‚0 disk.sda.out‚ 134.48046875 disk.drbd1289.out‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚17608.60107421875 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 network.sent‚ 4.877392e+06 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 134.48046875 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51317474662973% disk./.usageGB‚1260.8471794128418 disk.drbd1828.in‚0 network.recv‚323130 disk.sda2.in‚0.03125 disk.drbd1550.in‚0WüÔžý‚(:“ ¡ ŠÓ©¿Ä³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.837' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.61233333333334 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DqÖ=þ‚(:²  ¡ ŠÓ˶å³ disk.sda.out‚ 135.09375 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 135.09375 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.742973327636719 network.recv‚323280 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 62.1015625 cpu‚25.289455446626125 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51317474662973 disk.sda1.out‚0( proc.memory.availableMB‚ 1940.3203125 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚17608.78076171875$ memory_percent‚92.10481643676758 disk.rbd1.in‚0 disk.rbd1.out‚ 23.1982421875 network.sent‚ 4.877542e+06% disk./.usageGB‚1260.8471794128418 disk.drbd1828.out‚0# disk.dm-0.out‚17608.78076171875 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1584.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125! proc.memory.rssMB‚ 2885.953125 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.sda2.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd2118.out‚0í]ê2ÿ- _wandbruntime‚5091  _runtime‚5091À|›É€Ã _step‚229 ' lr-AdamW/pg1‚0.00010765404635462268 ' lr-AdamW/pg2‚0.00010765404635462268 trainer/global_step‚2799 % _timestamp‚1.7848443284245954e+09 _runtime‚5091.781836472Ë ûÎÈ _step‚229 ' lr-AdamW/pg1‚0.00010765404635462268 ' lr-AdamW/pg2‚0.00010765404635462268 trainer/global_step‚2799 % _timestamp‚1.7848443284245954e+09 _runtime‚5091.781836472åB\°Ô2‚- _wandbruntime‚5091  _runtime‚5091„¶NŸó'ƒíO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚806996786724864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚804080188049408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224240942302720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚230 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25965817 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24236896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚806998158334464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚804080161875968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2913085150444 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24236902 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24236896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25965817 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2917970285056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚806998158334464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2916624848896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7848443284256806e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚806996786724864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25955753 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3566558801408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚804080161875968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15961671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220674015410176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50192649 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3566507534336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚2799 _runtime‚5091.786943757 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚803762744655408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2917970285056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚365 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224240522944512 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806674485194566 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15962036 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50202719 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8503300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2916624848896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚806675855233820 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚804080188049408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803762770083376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2911740539158 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220674383501312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864Þµ™ï „òO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚230 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25965817 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24236896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚806998158334464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚804080161875968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2913085150444 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24236902 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24236896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25965817 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2917970285056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_||u bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚806998158334464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2916624848896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7848443284256806e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚806996786724864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25955753 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3566558801408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚804080161875968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15961671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220674015410176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50192649 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3566507534336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚2799 _runtime‚5091.786943757 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚803762744655408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2917970285056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚365 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224240522944512 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806674485194566 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15962036 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50202719 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8503300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2916624848896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚806675855233820 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚804080188049408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803762770083376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2911740539158 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220674383501312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚806996786724864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚804080188049408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224240942302720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005æŠê2…- _wandbruntime‚5093  _runtime‚5093Ín÷ý&†÷M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7461371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2914105952196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15967601 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8505869 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24245263 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚5093.262152099 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 % _timestamp‚1.7848443299045029e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3567804436992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚804041571857152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7461376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15967240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚807278090140160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚804359097339904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚804359071166464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚231 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50210098 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24245263 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚804041546429184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24245269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚807276718523392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2918992800256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220752969867264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚807278090140160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25964835 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50220182 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50220182 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50210098 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 Eÿ'‡üM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15967601 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8505869 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24245263 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚5093.262152099 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 % _timestamp‚1.7848443299045029e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3567804436992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚804041571857152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7461376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15967240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚807278090140160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚804359097339904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚804359071166464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚231 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50210098 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24245263 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚804041546429184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24245269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚807276718523392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2918992800256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220752969867264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚807278090140160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25964835 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50220182 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50220182 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50210098 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7461371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2914105952196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181ç›~V”2ˆ- _wandbruntime‚5093  _runtime‚50935Y‰ý _runtime‚5093.297698569 .loss/FRt1/assignment_loss‚2.07980736432286 /loss/FBt2/detection_loss‚0.5306774036303977 epoch‚9 0loss/FRt2/assignment_loss‚0.9611497670074168 1loss/SRqqt2/assignment_loss‚0.660315303677339 2loss/FBt1/assignment_loss‚0.020100290010741446 1loss/SRqqt1/detection_loss‚0.9822529572730245 2loss/SRqqt1/assignment_loss‚0.7679176136132251 1loss/FBt2/assignment_loss‚0.05866549939390037 /loss/FRt1/detection_loss‚0.6414222163127835 &loss/total_loss‚10.335513494641464 trainer/global_step‚2799 # _timestamp‚1.78484432994051e+09 _step‚232 .loss/FRt2/detection_loss‚1.554092019306424 1loss/SRqqt2/detection_loss‚0.6765517172281224 /loss/FBt1/detection_loss‚1.4025613428652286<| HŠ‚ 2loss/SRqqt1/assignment_loss‚0.7679176136132251 1loss/FBt2/assignment_loss‚0.05866549939390037 /loss/FRt1/detection_loss‚0.6414222163127835 &loss/total_loss‚10.335513494641464 trainer/global_step‚2799 # _timestamp‚1.78484432994051e+09 _step‚232 .loss/FRt2/detection_loss‚1.554092019306424 1loss/SRqqt2/detection_loss‚0.6765517172281224 /loss/FBt1/detection_loss‚1.4025613428652286 _runtime‚5093.297698569 .loss/FRt1/assignment_loss‚2.07980736432286 /loss/FBt2/detection_loss‚0.5306774036303977 epoch‚9 0loss/FRt2/assignment_loss‚0.9611497670074168 1loss/SRqqt2/assignment_loss‚0.660315303677339 2loss/FBt1/assignment_loss‚0.020100290010741446 1loss/SRqqt1/detection_loss‚0.9822529572730245èÕ .Ÿ‹‚(:” ° ŠÓ§ƒ¿³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.329' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.77633333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qîx»>Œ‚(:³  ° ŠÓ×Åݳ cpu‚25.292019237815694 disk.drbd1552.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.rbd1.out‚ 23.1982421875 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0! proc.memory.rssMB‚ 2885.953125* proc.memory.percent‚11.742973327636719 disk.drbd1673.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 network.recv‚323280 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 135.34375# disk.dm-0.out‚17609.36572265625 disk.drbd2041.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚17609.36572265625) proc.memory.availableMB‚ 1941.20703125 disk.sda2.out‚ 135.34375 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8471794128418 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1938.out‚0 network.sent‚ 4.877542e+06 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 62.1015625 disk.drbd1285.out‚0$ memory_percent‚92.10120836893717) disk./.usagePercent‚70.51317474662973 disk.drbd1226.in‚0^a\¢2- _wandbruntime‚5112  _runtime‚5112¹ $-Žâv 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2339506596326828 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 _runtime‚ 5112.70674547 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3945205509662628 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7125223278999329 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6792445182800293 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07165464013814926 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8508927226066589 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20432914793491364 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8956342339515686 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8508927226066589 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9982993006706238 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8263888359069824 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4154234528541565 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5618501901626587 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7797541618347168 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7678431868553162 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7462996244430542 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21473321318626404 9"PuriÇÑN;4ty/*FRt2SRqqt*FBt/SRqqt_purity‚0.7082844972610474 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7432989478111267 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21758243441581726 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.816986083984375 particle/accuracy_2_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.177101731300354 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974795579910278 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18194425106048584 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _step‚233 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7227074503898621 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3923506736755371 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.606433093547821 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2638888955116272 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7801149487495422 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7417458295822144 )jet/accuracy_1_of_3‚0.921412467956543 *jet/accuracy_2_of_3‚0.5635885000228882 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9817619323730469 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6961079835891724 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8904257416725159 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9993678331375122 /particle/accuracy_4_of_4‚0.9271352887153625 (particle/accuracy‚0.8481329083442688 $ _timestamp‚1.784844349348695e+09 )loss/val_total_loss‚9.985746446098528 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9942070841789246 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0361991710960865 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22236450016498566 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8090843558311462 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.019494129344820976 *jet/accuracy_1_of_2‚0.7829561829566956 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.997036337852478 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7474541664123535 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7366659045219421 particle/accuracy_1_of_1‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9908801317214966 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28625407814979553 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19063138961791992 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6405423879623413 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6576461791992188 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9752820134162903 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6652398705482483 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6149481534957886 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28279218077659607 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.653005063533783 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1467374563217163 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3315897583961487 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3055555522441864 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7909786105155945 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6776791214942932 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993788599967957 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6526775360107422 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8102623224258423 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.11570918560028076 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45071783661842346 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2364042103290558 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.52733314037323 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1904156655073166 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7061295509338379 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1448550671339035 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47309061884880066 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05937535688281059 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19285617768764496 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7041997313499451 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9985119104385376 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4435897469520569 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6989402770996094 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9747141599655151 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04307696223258972 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3205261826515198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472221493721008 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976958632469177 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22236450016498566 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04175099730491638 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6948717832565308 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9956600069999695 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03801197186112404 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6610470414161682 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6652398705482483 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6849412322044373 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24882793426513672 7validation_average_jet_accuracy‚0.37206894159317017 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12436343729496002 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5303030610084534 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05241462588310242 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6359136700630188 'particle/f_score‚0.6346820592880249 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2605562210083008 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43124398589134216 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5912942886352539 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36667218804359436 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40744367241859436 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6668134927749634 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8934492468833923 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13103535771369934 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06000825762748718 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2165907323360443 particle/accuracy_2_of_3‚1 +particle/specificity‚0.9118459820747375 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2217322289943695 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7015814185142517 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17991428077220917 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2027968019247055 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2638888955116272 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9817602038383484 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5956043601036072 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.554565966129303 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6392568349838257 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.98127281665802 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9983323812484741 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 +particle/sensitivity‚0.6148813366889954 trainer/global_step‚2799 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21449773013591766 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7907649278640747 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6455645561218262 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7450500130653381 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17224153876304626 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7668899893760681 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992431104183197 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23531095683574677 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04960516840219498 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04175099730491638 particle/accuracy_1_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3055555522441864 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45372483134269714 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24524657428264618 ,jet/accuracy_3_of_3‚0.060972776263952255 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05185208469629288 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9876682162284851 epoch‚9 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11124660819768906 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8277719616889954 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.530950665473938 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6543003916740417 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8473161458969116 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2081509828567505¸™qm;çv 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3055555522441864 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7909786105155945 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6776791214942932 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993788599967957 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6526775360107422 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8102623224258423 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.11570918560028076 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45071783661842346 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2364042103290558 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.52733314037323 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1904156655073166 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7061295509338379 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1448550671339035 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47309061884880066 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05937535688281059 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19285617768764496 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7041997313499451 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9985119104385376 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4435897469520569 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6989402770996094 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9747141599655151 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04307696223258972 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3205261826515198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472221493721008 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976958632469177 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22236450016498566 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04175099730491638 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6948717832565308 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9956600069999695 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03801197186112404 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6610470414161682 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6652398705482483 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6849412322044373 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24882793426513672 7validation_average_jet_accuracy‚0.37206894159317017 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12436343729496002 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5303030610084534 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05241462588310242 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6359136700630188 'particle/f_score‚0.6346820592880249 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2605562210083008 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43124398589134216 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5912942886352539 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36667218804359436 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40744367241859436 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6668134927749634 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8934492468833923 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13103535771369934 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06000825762748718 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2165907323360443 particle/accuracy_2_of_3‚1 +particle/specificity‚0.9118459820747375 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2217322289943695 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7015814185142517 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17991428077220917 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2027968019247055 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2638888955116272 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9817602038383484 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5956043601036072 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.554565966129303 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6392568349838257 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.98127281665802 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9983323812484741 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 +particle/sensitivity‚0.6148813366889954 trainer/global_step‚2799 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21449773013591766 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7907649278640747 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6455645561218262 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7450500130653381 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17224153876304626 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7668899893760681 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992431104183197 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23531095683574677 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04960516840219498 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04175099730491638 particle/accuracy_1_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3055555522441864 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45372483134269714 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24524657428264618 ,jet/accuracy_3_of_3‚0.060972776263952255 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05185208469629288 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9876682162284851 epoch‚9 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11124660819768906 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8277719616889954 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.530950665473938 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6543003916740417 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8473161458969116 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2081509828567505 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2339506596326828 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 _runtime‚ 5112.70674547 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3945205509662628 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7125223278999329 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6792445182800293 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07165464013814926 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8508927226066589 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20432914793491364 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8956342339515686 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8508927226066589 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9982993006706238 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8263888359069824 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4154234528541565 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5618501901626587 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7797541618347168 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7678431868553162 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7462996244430542 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21473321318626404 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7082844972610474 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7432989478111267 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21758243441581726 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.816986083984375 particle/accuracy_2_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.177101731300354 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974795579910278 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18194425106048584 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _step‚233 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7227074503898621 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3923506736755371 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.606433093547821 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2638888955116272 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7801149487495422 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7417458295822144 )jet/accuracy_1_of_3‚0.921412467956543 *jet/accuracy_2_of_3‚0.5635885000228882 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9817619323730469 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6961079835891724 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8904257416725159 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9993678331375122 /particle/accuracy_4_of_4‚0.9271352887153625 (particle/accuracy‚0.8481329083442688 $ _timestamp‚1.784844349348695e+09 )loss/val_total_loss‚9.985746446098528 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9942070841789246 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0361991710960865 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22236450016498566 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8090843558311462 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.019494129344820976 *jet/accuracy_1_of_2‚0.7829561829566956 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.997036337852478 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7474541664123535 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7366659045219421 particle/accuracy_1_of_1‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9908801317214966 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28625407814979553 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19063138961791992 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6405423879623413 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6576461791992188 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9752820134162903 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6652398705482483 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6149481534957886 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28279218077659607 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.653005063533783 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1467374563217163 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3315897583961487 particle/accuracy_1_of_3‚1éó‡ù£@‚(:µ  ¿ ŠÓÂÃݳ disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0$ memory_percent‚88.14169565836588 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.743640899658203 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 135.62109375 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.dm-0.in‚ 62.1015625& disk.nvme0n1.out‚17609.57666015625 disk.nvme0n1.in‚ 62.1015625 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 23.33203125 disk.sda2.out‚ 135.62109375 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2886.1171875 cpu‚25.298646190143604 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 network.sent‚ 4.942769e+06 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1510.out‚0 disk.drbd1552.out‚0' proc.memory.availableMB‚ 2914.296875 network.recv‚325052) disk./.usagePercent‚70.51338616443003$ disk./.usageGB‚1260.850959777832# disk.dm-0.out‚17609.57666015625 disk.drbd2078.in‚0 disk.drbd1584.out‚0³ee‘‘‚(:† ¿ ŠÓÆýº³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚58.0 gpu.0.powerWatts‚105.852' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…‘0-“’‚(:ˆ ΠŠÓ³£§³ gpu.0.gpu‚69.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚59.0 gpu.0.powerWatts‚107.316' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<Å¢Æ>“‚(:³  ΠŠÓ©òdz* proc.memory.percent‚11.743640899658203 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 62.1015625 disk.drbd2041.in‚0 network.recv‚325160 disk.drbd1510.in‚0 disk.sda.out‚ 136.015625 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 136.015625# memory_percent‚88.1416161855062 disk.drbd1510.out‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚18579.48876953125" proc.memory.rssMB‚ 2886.1171875 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 network.sent‚ 4.942877e+06 disk.drbd1828.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 62.1015625 disk.drbd2118.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0) disk./.usagePercent‚70.51338829780843 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8509979248047 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 cpu‚25.296231087365076 disk.rbd1.out‚ 23.33203125& disk.nvme0n1.out‚18579.48876953125) proc.memory.availableMB‚ 2914.31640625¾¹…”‚(:­  Ý ŠÓ´Òÿ³ disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.dWÞ§³rbd1550.out‚0" proc.memory.rssMB‚ 2886.1171875) disk./.usagePercent‚70.51338616443003 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.sda2.out‚136.4375 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0$ disk./.usageGB‚1260.850959777832 disk.drbd1226.in‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 2914.31640625* proc.memory.percent‚11.743640899658203 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0& disk.nvme0n1.out‚18579.74267578125 network.sent‚ 4.942943e+06 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2041.in‚0 network.recv‚325226 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0# memory_percent‚88.1416161855062# disk.dm-0.out‚18579.74267578125 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 23.39453125 cpu‚25.29623751922726 disk.drbd1584.in‚0 disk.sda.out‚136.4375 disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd1673.in‚0 disk.dm-0.in‚ 62.1015625? +¡Ÿ•‚(:” Ý ŠÓõÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚310.094' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.36466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß÷KŸ–‚(:” ì ŠÓõ ¸³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.701' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.56700000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ÂFë<—‚(:±  ì ŠÓôÏÕ³ disk.drbd1289.out‚0) disk./.usagePercent‚70.51336653734866# disk.dm-0.out‚19549.73486328125 network.sent‚ 4.943051e+06 disk.sda2.out‚ 136.71875 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 2914.28515625 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.dm-0.in‚ 62.1015625 disk.drbd1965.out‚0 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2886.1171875 disk.drbd1356.in‚0 network.recv‚325334 disk.drbd1673.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.743640899658203 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚19549.73486328125 disk.sda.out‚ 136.71875 disk.drbd2078.in‚0 disk.rbd1.out‚ 23.39453125 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0% disk./.usageGB‚1260.8506088256836 disk.sda.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1285.out‚0# memory_percent‚88.1417433420817 cpu‚25.296155478701003 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd1817.in‚0 disk.drbd1285.in‚0Þ¢•Çž˜‚(:“ û ŠÓØÅѳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚288.286' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.09533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð´n?™‚(:´  û ŠÓþô³) proc.memory.availableMB‚ 2914.28515625 disk.drbd1938.in‚0 disk.drbd1938.out‚0( disk./.usagePercent‚70.5133667506865 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚19549.94580078125 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 62.1015625 disk.drbd1552.in‚0 disk.drbd1931.out‚0 network.recv‚326729 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.743640899658203 disk.drbd1550.in‚0 disk.sda.out‚ 137.046875 disk.nvme0n1.in‚ 62.1015625 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2886.1171875 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚19549.94580078125 disk.drbd1673.out‚0 disk.drbd1584.in‚0 cpu‚25.296020053150887 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 network.sent‚ 5.008207e+06% disk./.usageGB‚1260.8506126403809 disk.drbd1285.in‚0# memory_percent‚88.1417433420817 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 23.5283203125 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.sda2.out‚ 137.046875¦ù¸žš‚(:“ Š¡ŠÓ¥”´³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.298' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.43266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆP9:›‚(:¯  Š¡ŠÓÕþѳ network.sent‚ 5.008315e+06 network.recv‚326837) disk./.usagePercent‚70.51336397729457 disk.drbd2078.in‚0 disk.rbd1.out‚ 23.5283203125 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0$ memory_percent‚92.09663073221842 disk.drbd2111.out‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8505630493164 disk.drbd2078.out‚0" disk.nvme0n1.out‚ 19550.6796875 disk.drbd1510.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1942.33203125 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd2118.out‚0 disk.sda.out‚ 137.3359375 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.743640899658203 disk.dm-0.out‚ 19550.6796875 disk.drbd1965.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 137.3359375 disk.dm-0.in‚ 62.1015625 cpu‚25.29593856019475 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2886.1171875 disk.drbd1938.in‚0 disk.drbd1356.out‚0&8$2œ- _wandbruntime‚5203  _runtime‚5203¼z¹Éà ' lr-AdamW/pg1‚0.00010728518334626226 ' lr-AdamW/pg2‚0.00010728518334626226 trainer/global_step‚2849 % _timestamp‚1.7848444397539358e+09 _runtime‚5203.111182322 _step‚234AšƒýÎžÈ ' lr-AdamW/pg1‚0.00010728518334626226 ' lr-AdamW/pg2‚0.00010728518334626226 trainer/global_step‚2849 % _timestamp‚1.7848444397539358e+09 _runtime‚5203.111182322 _step‚234êRO½P2Ÿ- _wandbruntime‚5203  _runtime‚5203VÕCó' íO _runtime‚5203.115966694 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51203695 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24744684 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26469081 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3636416561152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚820261859923344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7848444397549882e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2969797872566 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26459017 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 trainer/global_step‚2849 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚823561107579392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7597628 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚823559735969792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24744684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26469081 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224652701312000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24744678 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚820586329298944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2973432844288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8678649 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820586329298944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚823230287756656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _step‚235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚823231657795910 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26459017 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8678313 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228289117873152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚823559735969792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2973432844288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820586303125504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51213765 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚341 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224652333220864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2968453261280 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚823561107579392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7597623 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2974778280448 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968`£Û¦ø'¡òO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3636367391232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚823230287756656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _step‚235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚823231657795910 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26459017 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8678313 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228289117873152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚823559735969792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2973432844288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820586303125504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51213765 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚341 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224652333220864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2968453261280 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚823561107579392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7597623 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2974778280448 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚5203.115966694 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51203695 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24744684 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26469081 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3636416561152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚820261859923344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7848444397549882e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2969797872566 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26459017 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 trainer/global_step‚2849 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚823561107579392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7597628 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚823559735969792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24744684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26469081 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224652701312000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24744678 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚820586329298944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2973432844288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8678649 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820586329298944 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51221144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228368951698432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16281818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3637606469120 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24753051 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2969474062920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224730919586816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7848444412378893e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚341 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820865238589440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚820865212416000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51231228 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24753045 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚5204.595552483 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51231228 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7600268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26468099 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2970818674318 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26468099 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8681214 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚820865212416000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚823511480371438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2974455352320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚236 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26478177 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3637664020480 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚820865238589440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16281477 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228368526055936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚823510110332072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚820540636269152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26478177 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24753051 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚823841039385088 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24753045 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0¤n ë'¤üM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚820865238589440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16281477 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228368526055936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚823510110332072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚820540636269152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26478177 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24753051 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚823841039385088 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24753045 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51221144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228368951698432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16281818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3637606469120 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24753051 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2969474062920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224730919586816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7848444412378893e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚341 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820865238589440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚820865212416000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51231228 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24753045 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚5204.595552483 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51231228 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7600268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26468099 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2970818674318 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26468099 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8681214 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚820865212416000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚823511480371438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2974455352320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚236 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26478177 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3637664020480 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360ìÝ+ 92¥- _wandbruntime‚5204  _runtime‚5204¶èΦþ trainer/global_step‚2849 0loss/FRt2/assignment_loss‚0.9045304565718679 1loss/SRqqt1/assignment_loss‚0.713075436002452 /loss/FRt2/detection_loss‚1.4995358488441837 /loss/SRqqt2/detection_loss‚0.67798909334455 /loss/FBt1/detection_loss‚1.0073447717974584 /loss/FBt2/detection_loss‚0.4740751364122153 0loss/FBt2/assignment_loss‚0.0805185140199277 /loss/FRt1/detection_loss‚0.6005098405061406 _runtime‚5204.625905385 0loss/FRt1/assignment_loss‚2.0919841512932638 2loss/SRqqt2/assignment_loss‚0.5692952107901212 1loss/FBt1/assignment_loss‚0.05498119272193788 1loss/SRqqt1/detection_loss‚1.1620070729181426 %loss/total_loss‚9.835846725222261 epoch‚10 % _timestamp‚1.7848444412686872e+09 _step‚237¤óý‡ §ƒ 0loss/FRt2/assignment_loss‚0.9045304565718679 1loss/SRqqt1/assignment_loss‚0.713075436002452 /loss/FRt2/detection_loss‚1.4995358488441837 /loss/SRqqt2/detection_loss‚0.67798909334455 /loss/FBt1/detection_loss‚1.0073447717974584 /loss/FBt2/detection_loss‚0.4740751364122153 0loss/FBt2/assignment_loss‚0.0805185140199277 /loss/FRt1/detection_loss‚0.6005098405061406 _runtime‚5204.625905385 0loss/FRt1/assignment_loss‚2.0919841512932638 2loss/SRqqt2/assignment_loss‚0.5692952107901212 1loss/FBt1/assignment_loss‚0.05498119272193788 1loss/SRqqt1/detection_loss‚1.1620070729181426 %loss/total_loss‚9.835846725222261 epoch‚10 % _timestamp‚1.7848444412686872e+09 _step‚237 trainer/global_step‚2849í&©®ž¨‚(:“ ™¡ŠÓ—¸«³ gpu.0.gpu‚47.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.99' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.33000000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÓ‘ø=©‚(:²  ™¡ŠÓèœÐ³ disk.sda.out‚ 137.78515625 disk.drbd1673.out‚0 disk.sda2.out‚ 137.78515625 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.743640899658203 network.sent‚ 5.008381e+06 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 23.5908203125 disk.drbd1285.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.09977785746256 disk.dm-0.in‚ 62.1015625 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2886.1171875 disk.dm-0.out‚19550.91015625 disk.drbd1510.out‚0# disk./.usageGB‚1260.85054397583 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 62.1015625 disk.sda1.in‚0 network.recv‚326903 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 cpu‚25.295998050240755) disk./.usagePercent‚70.51336291060537 disk.drbd2118.in‚0) proc.memory.availableMB‚ 1941.55859375 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0# disk.nvme0n1.out‚19550.91015625 disk.sda1.out‚0I'âKžª‚(:“ ¨¡ŠÓ—Ðʳ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.026' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.00866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØßÒB«‚(:·  ¨¡ŠÓ´îë³) disk./.usagePercent‚70.51336291060537 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚19551.42919921875 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.743640899658203 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 cpu‚25.29605488913253 disk.drbd1938.in‚0# disk.dm-0.out‚19551.42919921875) proc.memory.availableMB‚ 1941.47265625" proc.memory.rssMB‚ 2886.1171875 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1828.in‚0 network.recv‚327011 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.dm-0.in‚ 62.1015625 disk.nvme0n1.in‚ 62.1015625 disk.drbd1226.in‚0$ memory_percent‚92.10012753804526# disk./.usageGB‚1260.85054397583 disk.sda.out‚ 138.06640625 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 23.6845703125 disk.sda2.in‚0.03125 disk.sda2.out‚ 138.06640625 disk.drbd1673.in‚0 network.sent‚ 5.008489e+06Øû:ž¬‚(:“ ·¡ŠÓï´³ gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.227' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.74233333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â~B­‚(:·  ·¡ŠÓŒ¤×³ disk.drbd1356.out‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.743640899658203 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 138.3671875 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0" proc.memory.rssMB‚ 2886.1171875 disk.sda.out‚ 138.3671875 disk.drbd2041.out‚0& disk.nvme0n1.out‚19551.80810546875 network.sent‚ 5.073645e+06 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 23.8486328125% disk./.usageGB‚1260.8510513305664) proc.memory.availableMB‚ 1941.38671875 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 cpu‚25.295972817478596 disk.drbd1550.out‚0# disk.dm-0.out‚19551.80810546875 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.10047721862793 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 network.recv‚328320 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 62.1015625 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda2.in‚0.03125( disk./.usagePercent‚70.5133912845382 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0P§Ã®‚(:’ Æ¡ŠÓ·à³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚110.48' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.82666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á@¦$8¯‚(:­  Æ¡ŠÓ¾ªÿ³ disk.drbd1552.out‚0 disk.sda.out‚ 138.59375 disk.drbd1673.in‚0 disk.drbd1673.out‚0" proc.memory.rssMB‚ 2886.1171875 disk.drbd2078.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 network.recv‚329968 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125$ memory_percent‚92.10087458292642 disk.dm-0.in‚ 62.1015625 cpu‚25.295981118998004 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0# disk./.usageGB‚1260.85103225708 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0( proc.memory.availableMB‚ 1941.2890625 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 24.2080078125 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.743640899658203 disk.drbd1289.out‚0 disk.drbd1938.in‚0" disk.dm-0.out‚19552.2060546875 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 network.sent‚ 5.141447e+06' disk./.usagePercent‚70.513390217849 disk.nvme0n1.in‚ 62.1015625% disk.nvme0n1.out‚19552.2060546875 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 138.59375ï k3?°‚(:´  Õ¡ŠÓìÆɳ network.recv‚330034 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0$ memory_percent‚92.09985733032227 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.sda.out‚ 139.1015625 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1289.in‚0" disk.dm-0.out‚19552.4091796875 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2886.1171875 network.sent‚ 5.141513e+06 disk.sda2.out‚ 139.1015625 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.rbd1.out‚ 24.2705078125 cpu‚25.295807972074734 disk.drbd1356.in‚0 disk.drbd1226.out‚0( disk./.usagePercent‚70.5133912845382 disk.drbd1584.in‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.743640899658203 disk.drbd2078.in‚0% disk./.usageGB‚1260.8510513305664 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.dm-0.in‚ 62.1015625 disk.drbd1828.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚19552.4091796875 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1941.5390625 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0uìo)“±‚(:ˆ Õ¡ŠÓ²¶©³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚269.037' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.679 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05[ÓqŸ²‚(:” ä¡ŠÓ…À½³ gpu.0.gpu‚41.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.675' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.55833333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õF9<³‚(:±  ä¡ŠÓªÀà³ disk.drbd1938.out‚0 disk.rbd1.out‚ 24.2705078125 network.sent‚ 5.141579e+06" disk.dm-0.out‚19552.8134765625 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 62.1015625 cpu‚25.29586506993557 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 1941.29296875 disk.drbd1356.out‚0' disk./.usagePercent‚70.513390217849 disk.drbd2041.out‚0% disk.nvme0n1.out‚19552.8134765625 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 139.4921875 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0# disk./.usageGB‚1260.85103225708 disk.drbd1828.out‚0 disk.sda1.in‚0 network.recv‚330100 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 139.4921875$ memory_percent‚92.10085868835449* proc.memory.percent‚11.743640899658203 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 62.1015625 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125" proc.memory.rssMB‚ 2886.1171875 disk.drbd2078.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17RƇ 2´- _wandbruntime‚5290  _runtime‚5290Ï‘!šÉµÃ ' lr-AdamW/pg1‚0.00010690993417407126 ' lr-AdamW/pg2‚0.00010690993417407126 trainer/global_step‚2899 % _timestamp‚1.7848445267622178e+09 _runtime‚5290.119454757 _step‚238«¹áHÎ¶È % _timestamp‚1.7848445267622178e+09 _runtime‚5290.119454757 _step‚238 ' lr-AdamW/pg1‚0.00010690993417407126 ' lr-AdamW/pg2‚0.00010690993417407126 trainer/global_step‚2899î&±F2·- _wandbruntime‚5290  _runtime‚5290L ¦¸ìO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3020901755374 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52077997 YEDevic¶³LeStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚837557761660416 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228581651518464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3698859670528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26914969 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25163034 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25163028 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26914969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚837556390050816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚837556390050816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52088067 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3024622400512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834531793823744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834531767650304 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25163028 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚837221480328264 $ _timestamp‚1.784844526763283e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3024622400512 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26925033 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8825155 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834531767650304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚837557761660416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7729628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3025967836672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232280455727616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚834201923184176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2899™hÁÖ÷'¹ñO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25163034 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25163028 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26914969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚837556390050816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚837556390050816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52088067 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3024622400512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834531793823744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834531767650304 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25163028 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚837221480328264 $ _timestamp‚1.784844526763283e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3024622400512 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26925033 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8825155 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834531767650304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚837557761660416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7729628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3025967836672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232280455727616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚834201923184176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2899 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3020901755374 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52077997 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚837557761660416 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228581651518464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3698859670528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26914969 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653ïtN2*2º- _wandbruntime‚5291  _runtime‚5291•ñiû&»õM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52105530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228660237884416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3700108113920 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16560373 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚837837693466112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232360714089472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52105530 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚834810703114240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3025644908544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26934129 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3021922557126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3020577945728 _step‚240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3700058951168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8828461 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8828110 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834810703114240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5291.598847464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚837837693466112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26924051 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623GËŒçm¼úM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52095446 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7732263 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 # _timestamp‚1.78484452824119e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26924051 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚834810676940800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚834810676940800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25171395 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52105530 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚834810703114240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3025644908544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26934129 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3021922557126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3020577945728 _step‚240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3700058951168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8828461 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8828110 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834810703114240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5291.598847š¯pË“464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚837837693466112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26924051 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52105530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228660237884416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3700108113920 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16560373 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚837837693466112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232360714089472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚851553044131840 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚2949 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52962369 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16833693 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8972065 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851212672899872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3761374508544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7861628 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25581384 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27370921 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3761325338624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27380985 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25581384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚851553044131840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27380985 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851214042939126 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚848477232175104 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚851554415741440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27370921 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0µöxq ÑòO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7861628 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25581384 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27370921 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3761325338624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27380985 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25581384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚851553044131840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27380985 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851214042939126 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚848477232175104 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.currenŒ×Mчt‚824 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚851554415741440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27370921 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236272295154688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3072005638182 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25581378 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3075811956736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8971713 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 % _timestamp‚1.7848446137483716e+09 _step‚243 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232511337907200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848477258348544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚5377.108932806 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚848477258348544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3077157392896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7861623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚848142037300944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3070661026896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16833336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚851553044131840 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚2949 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52962369 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16833693 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8972065 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851212672899872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3761374508544óœq´2Ò- _wandbruntime‚5378  _runtime‚5378”÷O{ù&ÓóM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚244 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8974957 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3762569133056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3078179908096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7864268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚851832975930368 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16838867 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 5378.5873432 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27390081 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25589751 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52979832 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚851493865514654 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚848420839074720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52969748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚851492495475288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52979832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232589924273152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚848756141465600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3076834464768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚848420813646752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3071681828536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚851834347547136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52969748 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16839225 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112¦‡œYþ&ÔøM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16838867 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 5378.5873432 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27390081 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25589751 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52979832 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚851493865514654 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚848420839074720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52969748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚851492495475288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52979832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232589924273152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚848756141465600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3076834464768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚848420813646752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3071681828536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚851834347547136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52969748 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16839225 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚244 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8974957 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3762569133056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3078179908096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7864268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚851832975930368ô„]®2Õ- _wandbruntime‚5378  _runtime‚5378˜Âü`Öþ 0loss/FRt1/assignment_loss‚1.9716472836136232 1loss/SRqqt1/assignment_loss‚0.724461829094666 1loss/FBt2/assignment_loss‚0.04621132199901852 /loss/FRt1/detection_loss‚0.5769189099078145 1loss/SRqqt2/detection_loss‚0.6268991441749044 0loss/FRt2/assignment_loss‚0.8812658614025707 1loss/FBt1/assignment_loss‚0.03769956063597772 /loss/FBt1/detection_loss‚1.3495980519159088 /loss/FBt2/detection_loss‚0.6479318149380843 %loss/total_loss‚10.16821798740055 epoch‚10 _runtime‚5378.620675085 _step‚245 2loss/SRqqt2/assignment_loss‚0.5861599528993002 % _timestamp‚1.7848446152634761e+09 /loss/FRt2/detection_loss‚1.5116378116025795 .loss/SRqqt1/detection_loss‚1.2077864452161 trainer/global_step‚2949"1R? ׃ /loss/FRt2/detection_loss‚1.5116378116025795 .loss/SRqqt1/detection_loss‚1.2077864452161 trainer/global_step‚2949 0loss/FRt1/assignment_loss‚1.9716472836136232 1loss/SRqqt1/assignment_loss‚0.724461829094666 1loss/FBt2/assignment_loss‚0.04621132199901852 /loss/FRt1/detection_loss‚0.5769189099078145 1loss/SRqqt2/detection_loss‚0.6268991441749044 0loss/FRt2/assignment_loss‚0.8812658614025707 1loss/FBt1/assignment_loss‚0.03769956063597772 /loss/FBt1/detection_loss‚1.3495980519159088 /loss/FBt2/detection_loss‚0.6479318149380843 %loss/total_loss‚10.16821798740055 epoch‚10 _runtime‚5378.620675085 _step‚245 2loss/SRqqt2/assignment_loss‚0.5861599528993002 % _timestamp‚1.7848446152634761e+09õ ì1žØ‚(:“ Í¢ŠÓÅ—Á³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚295.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.49900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HËlEÙ‚(:º  Í¢ŠÓÎÞ³ disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 1940.33203125) disk./.usagePercent‚70.51336397729457 disk.drbd1938.in‚0& disk.nvme0n1.out‚19556.56982421875 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚19556.56982421875 disk.drbd1817.out‚0* proc.memory.percent‚11.743640899658203 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 network.sent‚ 5.308644e+06 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 141.80859375 disk.drbd1673.out‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2886.1171875 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 24.6630859375 disk.nvme0n1.in‚ 62.7421875 disk.drbd1226.in‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 62.7421875% disk./.usageGB‚1260.8505630493164 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 141.80859375 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 network.recv‚335771 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0$ memory_percent‚92.10476875305176 cpu‚25.295437848188342@«_“œÚ‚(:‘ Ü¢ŠÓþŒÄ³ gpu.0.gpu‚4.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚109.681' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.56033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó˜Ç>EÛ‚(:³  Ü¢ŠÓÞ„â³ disk.drbd2078.out‚0 disk.nvme0n1.in‚ 62.7421875 disk.drbd1550.out‚0 disk.sda.out‚ 142.29296875 disk.sda2.in‚0.03125% disk./.usageGB‚1260.8505630493164" disk.dm-0.out‚19556.9677734375 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 142.29296875 disk.drbd1289.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51336397729457 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125$ memory_percent‚92.10484822591147* proc.memory.percent‚11.743640899658203 disk.rbd1.out‚ 24.6630859375 disk.drbd1226.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 network.recv‚335837 disk.drbd1817.in‚0 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2886.1171875 disk.drbd1226.out‚0 disk.drbd1550.in‚0 network.sent‚ 5.30871e+06 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.iÅÛCÞùn‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 cpu‚25.295404150185934 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0% disk.nvme0n1.out‚19556.9677734375 disk.drbd1510.out‚0% proc.memory.availableMB‚ 1940.3125 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 62.7421875 disk.drbd2041.in‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17ã vh6Ü‚(:«  뢊Ó¸Îͳ disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 62.7421875 disk.nvme0n1.in‚ 62.7421875) disk./.usagePercent‚70.51336397729457 disk.drbd1931.out‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8505630493164 disk.sda.out‚ 142.65625 disk.drbd1289.in‚0" disk.dm-0.out‚19557.3037109375" proc.memory.rssMB‚ 2886.1171875 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚19557.3037109375 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0& proc.memory.availableMB‚ 1940.28125 disk.sda2.out‚ 142.65625 disk.drbd2041.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.743640899658203 cpu‚25.295280701750524 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 24.796875 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 network.recv‚336992 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1510.in‚0 network.sent‚ 5.316952e+06 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1584.in‚0$ memory_percent‚92.10497538248697Z zAŸÝ‚(:” 뢊ÓÀ×®³ gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.699' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.89966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èEþ“Þ‚(:ˆ ú¢ŠÓøËƳ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚172.566' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚57.522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õñý‘8ß‚(:­  ú¢ŠÓ«‘å³ disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 143.015625 disk.drbd1226.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51336504398378 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0" disk.dm-0.out‚19557.5712890625 disk.drbd2041.in‚0 disk.rbd1.out‚ 24.796875 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 62.7421875" proc.memory.rssMB‚ 2886.1171875 disk.drbd2118.in‚0 disk.sda.out‚ 143.015625 disk.drbd1510.out‚0* proc.memory.percent‚11.743640899658203 cpu‚25.295200372325024 network.sent‚ 5.317018e+06% disk./.usageGB‚1260.8505821228027 disk.drbd1552.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 network.recv‚337058 disk.drbd1584.in‚0 disk.rbd1.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0& proc.memory.availableMB‚ 1940.28125 disk.dm-0.in‚ 62.7421875 disk.drbd1931.in‚0 disk.drbd1965.out‚0$ memory_percent‚92.10497538248697 disk.drbd2118.out‚0 disk.drbd1584.out‚0% disk.nvme0n1.out‚19557.5712890625 disk.drbd1226.in‚0´Ä)Ãà‚(:’ ‰£ŠÓƾ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.49' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.83000000000001 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀÔ-)>á‚(:³  ‰£ŠÓÃõݳ disk.drbd1550.in‚0 disk.sda.out‚ 143.31640625 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 cpu‚25.295077170121477 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1940.27734375 network.sent‚ 5.382171e+06 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 62.7421875 disk.drbd1356.in‚0$ disk./.usageGB‚1260.848648071289 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125% disk.nvme0n1.out‚19557.7587890625 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51325688169847 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda2.out‚ 143.31640625 disk.drbd2111.in‚0$ memory_percent‚92.10499127705893 disk.rbd1.out‚ 24.859375 disk.drbd1289.in‚0" disk.dm-0.out‚19557.7587890625* proc.memory.percent‚11.743640899658203 disk.drbd1828.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 62.7421875 disk.drbd1673.in‚0 network.recv‚338834 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1550.out‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2886.1171875 proc.cpu.threads‚17UÅ”â‚(:‰ ˜£ŠÓÎø·³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚308.646' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ê8ã‚(:­  ˜£ŠÓ“óس disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.743640899658203 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 cpu‚25.295090983496625 disk.sda.out‚ 143.6953125 disk.dm-0.in‚ 62.7421875 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 62.7421875 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 24.859375" proc.memory.rssMB‚ 2886.1171875$ memory_percent‚92.09817250569662 disk.sda.in‚0.03125 disk.drbd1938.out‚0! disk.dm-0.out‚19558.087890625 disk.drbd1550.out‚0 network.sent‚ 5.382237e+06 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0' proc.memory.availableMB‚ 1941.953125 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1226.out‚0 network.recv‚338900 disk.drbd1931.out‚0$ disk.nvme0n1.out‚19558.087890625 disk.drbd1226.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.sda2.out‚ 143.6953125 disk.drbd1965.in‚0 disk.drbd1550.in‚0$ disk./.usageGB‚1260.848648071289 disk.drbd2078.out‚0) disk./.usagePercent‚70.51325688169847 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0‹û%o2ä- _wandbruntime‚5464  _runtime‚5464BJKÉåà ' lr-AdamW/pg1‚0.00010614047491510653 ' lr-AdamW/pg2‚0.00010614047491510653 trainer/global_step‚2999 % _timestamp‚1.7848447009301345e+09 _runtime‚5464.287829957 _step‚246™¾t£ÎæÈ ' lr-AdamW/pg2‚0.00010614047491510653 trainer/global_step‚2999 % _timestamp‚1.7848447009301345e+09 _runtime‚5464.287829957 _step‚246 ' lr-AdamW/pg1‚0.00010614047491510653öÿŒ¼%2ç- _wandbruntime‚5464  _runtime‚5464QYE«ò'èìO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚247 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3121764909704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚862422722873344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53836671 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚862422696699904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚865203865471480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3123109520990 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚367 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25999734 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25999734 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚862422696699904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17113066 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚865205235510734 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚865549698212864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚ 5464.28831224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7993623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236440288113664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25999728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27836937 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚865549698212864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3127001512960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3127001512960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9119443 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27826873 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9119810 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3823777503744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3823722042368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53826601 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚865551069822464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286XN¦¤÷'éñO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚862422696699904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚865203865471480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3123109520990 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚367 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25999734 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25999734 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚862422696699904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17113066 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚865205235510734 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚865549698212864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚ 5464.28831224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7993623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236440288113664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25999728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27836937 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚865549698212864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3127001512960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3127001512960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9119443 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27826873 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9119810 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3823777503744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3823722042368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53826601 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚865551069822464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚247 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3121764909704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚862422722873344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53836671 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824÷­sI2ê- _wandbruntime‚5465  _runtime‚5465ðÜÃE3 ëöM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3128024020992 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26008095 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚862360902335552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9122364 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26008095 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865831001628160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26008101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9122735 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27835955 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53854134 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7996268 ]KDBN{ŽÉeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3824962480128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53854134 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53844050 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865829630011392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 $ _timestamp‚1.784844702407167e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3129369464320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27846033 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3122785711344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26008101 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3129369464320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17119003 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7996263 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5465.771422918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240344264699392 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862360927763520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚865485058086262 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚865831001628160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27835955 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚862701605990400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚862701632163840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚862701605990400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3825022128640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚865483688046896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17118627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 trainer/global_step‚2999 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3124130322742 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824”{'ìûM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53854134 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53844050 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865829630011392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 $ _timestamp‚1.784844702407167e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3129369464320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27846033 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3122785711344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26008101 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3129369464320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17119003 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7996263 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5465.771422918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240344264699392 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862360927763520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚865485058086262 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚865831001628160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27835955 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚862701605990400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚862701632163840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚862701605990400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3825022128640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚865483688046896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17118627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 trainer/global_step‚2999 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3124130322742 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3128024020992 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26008095 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚862360902335552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9122364 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26008095 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865831001628160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26008101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9122735 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27835955 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53854134 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7996268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3824962480128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366øV²úS2í- _wandbruntime‚5465  _runtime‚5465æî¹î€ 0loss/FBt1/assignment_loss‚0.0372589114149773 1loss/FBt2/assignment_loss‚0.09431755396123503 /loss/FRt1/detection_loss‚0.5684585815601895 0loss/SRqqt2/detection_loss‚0.709614442279039 &loss/total_loss‚10.783065485720032 0loss/FRt1/assignment_loss‚1.9913592352572898 2loss/SRqqt2/assignment_loss‚0.6058516486985562 .loss/FRt2/detection_loss‚1.632696675855913 /loss/FBt2/detection_loss‚1.0018802829395723 epoch‚10 % _timestamp‚1.7848447024443896e+09 _step‚249 0loss/FRt2/assignment_loss‚0.9262451688751919 2loss/SRqqt1/assignment_loss‚0.6895968813678719 trainer/global_step‚2999 1loss/SRqqt1/detection_loss‚1.0529989785550593 /loss/FBt1/detection_loss‚1.4727871249551359 _runtime‚5465.801625751Õ: ï… _step‚249 0loss/FRt2/assignment_loss‚0.9262451688751919 2loss/SRqqt1/assignment_loss‚0.6895968813678719 trainer/global_step‚2999 1loss/SRqqt1/detection_loss‚1.0529989785550593 /loss/FBt1/detection_loss‚1.4727871249551359 _runtime‚5465.801625751 0loss/FBt1/assignment_loss‚0.0372589114149773 1loss/FBt2/assignment_loss‚0.09431755396123503 /loss/FRt1/detection_loss‚0.5684585815601895 0loss/SRqqt2/detection_loss‚0.709614442279039 &loss/total_loss‚10.783065485720032 0loss/FRt1/assignment_loss‚1.9913592352572898 2loss/SRqqt2/assignment_loss‚0.6058516486985562 .loss/FRt2/detection_loss‚1.632696675855913 /loss/FBt2/detection_loss‚1.0018802829395723 epoch‚10 % _timestamp‚1.7848447024443896e+09ù#Ñd”ð‚(:‰ §£ŠÓŒÿ·³ gpu.0.gpu‚68.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.336' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.112 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ÖÄ‘@ñ‚(:µ  §£ŠÓµšÙ³ disk.sda2.out‚ 143.95703125 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0$ memory_percent‚91.78150494893393 disk.drbd1552.out‚0 disk.drbd1828.out‚0$ disk.nvme0n1.out‚19558.443359375 network.sent‚ 5.388028e+06 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 cpu‚25.29524050479547 disk.drbd1550.in‚0* proc.memory.percent‚11.387157440185547" proc.memory.rssMB‚ 2798.5078125 disk.drbd1289.out‚0! disk.dm-0.out‚19558.443359375 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8476295471191 network.recv‚340034 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 62.7421875 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 2019.77734375) disk./.usagePercent‚70.51319992049497 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 25.0791015625 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 62.7421875 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.sda.out‚ 143.95703125ÈH<à“ò‚(:ˆ ¶£ŠÓüÊ»³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚277.833' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.611 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cˆ¯S=ó‚(:²  ¶£ŠÓýçÙ³ disk.drbd2041.out‚0$ disk.nvme0n1.out‚19559.337890625 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 144.16015625 disk.drbd1510.out‚0* proc.memory.percent‚11.387157440185547 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0# memory_percent‚91.7822519938151& proc.memory.availableMB‚ 2019.59375" proc.memory.rssMB‚ 2798.5078125 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 62.74609375( disk./.usagePercent‚70.5133119228614 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 62.74609375 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0 cpu‚25.29511756443666 proc.cpu.threads‚17% disk./.usageGB‚1260.8496322631836 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 network.sent‚ 5.388094e+06 disk.drbd1584.in‚0 disk.drbd1673.in‚0! disk.dm-0.out‚19559.337890625 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 25.0791015625 disk.sda.out‚ 144.16015625 disk.drbd1931.out‚0 disk.drbd1965.out‚0 network.recv‚340100 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚01ë©C<ô‚(:±  Å£ŠÓœîñ³ disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 25.1416015625 disk.drbd1552.out‚0* proc.memory.percent‚11.387157440185547 network.sent‚ 5.38816e+06 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0% disk./.usageGB‚1260.8496131896973 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0$ disk.nvme0n1.out‚19559.580078125 disk.drbd1226.in‚0 disk.sda2.out‚ 144.6015625 disk.drbd2041.in‚0 disk.drbd1550.out‚0 network.recv‚340166 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 cpu‚25.295038718367984 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51331085617221 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1285.in‚0# memory_percent‚91.7822519938151 disk.dm-0.in‚ 62.74609375 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0! disk.dm-0.out‚19559.580078125 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 144.6015625 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 62.74609375 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2798.5078125 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125& proc.memory.availableMB‚ 2019.59375 proc.cpu.threads‚17ÆGtΔõ‚(:‰ Å£ŠÓ†œÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚309.225' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚103.075 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ~}ö‚(:’ Ô£ŠÓáÁ¯³ gpu.0.gpu‚9.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.919' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.306333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vt~=A÷‚(:¶  Ô£ŠÓàÁÒ³ disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 25.2353515625% disk.nvme0n1.out‚19559.9580078125 disk.drbd1510.in‚0& proc.memory.availableMB‚ 2019.59375 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51330936280732 disk.drbd2118.out‚0 disk.sda2.out‚ 144.90234375 disk.drbd1356.out‚0 disk.drbd1226.out‚0 network.recv‚341597 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd2111.out‚0# memory_percent‚91.7822519938151 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0" proc.memory.rssMB‚ 2798.5078125* proc.memory.percent‚11.387157440185547 network.sent‚ 5.450722e+06 disk.drbd1552.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚ 144.90234375 disk.drbd1938.in‚0" disk.dm-0.out‚19559.9580078125 disk.drbd2041.out‚0% disk./.usageGB‚1260.8495864868164 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 62.74609375 disk.nvme0n1.in‚ 62.74609375 cpu‚25.294919402799376ac l?ø‚(:´  㣊Óºø‡´ disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 cpu‚25.294884234127654 network.recv‚341663% disk./.usageGB‚1260.8495864868164 disk.sda2.out‚ 145.3671875" disk.dm-0.out‚19560.1416015625* proc.memory.percent‚11.387157440185547 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚19560.1416015625 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 62.74609375 disk.drbd1226.out‚0# memory_percent‚91.7822519938151 proc.cpu.threads‚17 disk.drbd1510.out‚0) disk./.usagePercent‚70.51330936280732 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 network.sent‚ 5.450788e+06 disk.sda1.out‚0 disk.drbd1550.in‚0& proc.memory.availableMB‚ 2019.59375 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.dm-0.in‚ 62.74609375 disk.rbd1.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2798.5078125 disk.drbd1817.out‚0 disk.sda.out‚ 145.3671875 disk.rbd1.out‚ 25.3994140625 disk.drbd1285.in‚0´ŸGžù‚(:“ 㣊ÓÆÚå³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.148' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.04933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D§>žú‚(:“ ò£ŠÓÆܱ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.588' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.86266666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Ì¿G9û‚(:®  ò£ŠÓ‡½Ñ³ disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚19560.7431640625 disk.drbd1584.out‚0# memory_percent‚91.7822519938151 disk.drbd1828.out‚0 disk.rbd1.out‚ 25.3994140625" disk.dm-0.out‚19560.7431640625 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0" proc.memory.rssMB‚ 2798.5078125 network.sent‚ 5.4589e+06% disk./.usageGB‚1260.8506050109863 disk.drbd1552.out‚0& proc.memory.availableMB‚ 2019.59375 disk.sda.out‚ 145.65625 cpu‚25.294946782277325 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 62.74609375 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 network.recv‚342514) disk./.usagePercent‚70.51336632401082 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 145.65625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 62.74609375 disk.sda1.in‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.387157440185547¯±DK2ü- _wandbruntime‚5551  _runtime‚5551MßþØÉýà ' lr-AdamW/pg1‚0.00010574636568681243 ' lr-AdamW/pg2‚0.00010574636568681243 trainer/global_step‚3049 % _timestamp‚1.7848447878467078e+09 _runtime‚5551.204903217 _step‚250,P¸›ÎþÈ ' lr-AdamW/pg1‚0.00010574636568681243 ' lr-AdamW/pg2‚0.00010574636568681243 trainer/global_step‚3049 % _timestamp‚1.7848447878467078e+09 _runtime‚5551.204903217 _step‚250úÛÆÝ2ÿ- _wandbruntime‚5551  _runtime‚5551δ[uX€íO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28292889 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9267450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240369606411264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28282825 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚876368161224704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879546352293888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28282825 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879546352293888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8125628 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244255682170368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚3049 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3174213403798 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54700903 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26418078 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17393457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3178191069184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3886075759104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879195058043088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876368187398144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26418078 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚876368161224704 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8125623 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚379 % _timestamp‚1.7848447878474994e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54710973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879196428082342 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879547723903488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240369974502400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879547723903488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3179536505344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172868792512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876368187398144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17393073 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9267829 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876022214678544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876022189250576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26418084 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3179536505344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824¥ì¢@ø'òO % _timestamp‚1.7848447878474994e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54710973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879196428082342 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879547723903488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240369974502400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879547723903488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3179536505344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172868792512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876368187398144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17393073 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9267829 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876022214678544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876022189250576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26418084 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3179536505344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28292889 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9267450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240369606411264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28282825 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚876368161224704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879546352293888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28282825 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879546352293888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8125628 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244255682170368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚3049 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3174213403798 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54700903 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26418078 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17393457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3178191069184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3886075759104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879195058043088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876368187398144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26418078 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚876368161224704 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8125623 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚379û/ûß2‚- _wandbruntime‚5552  _runtime‚5552k/CÖý&ƒ÷M \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3180559020544 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240448192777216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28301985 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17398635 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26426451 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54728436 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚876647070515200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3179213577216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26426445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3887383701504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚879474880618504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9270751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28291907 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244335944569856 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54718352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28301985 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚3049 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54718352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244335516830208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879827655709184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3179213577216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26426451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚879476250657870 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8128268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879827655709184 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976ÒìÊÄüM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚876647096688640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3179213577216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26426445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3887383701504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚879474880618504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9270751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28291907 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244335944569856 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54718352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28301985 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚3049 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54718352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244335516830208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879827655709184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freè(Hçed‚ 3179213577216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26426451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚879476250657870 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8128268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879827655709184 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3180559020544 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240448192777216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28301985 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17398635 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26426451 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54728436 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚876647070515200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7552627921104431 *jet/accuracy_1_of_2‚0.7682105898857117 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26840507984161377 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1948956549167633 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.886854350566864 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22679026424884796 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7424242496490479 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915882349014282 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ,jet/accuracy_3_of_3‚0.059080176055431366 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9830737709999084 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7972584962844849 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 /particle/accuracy_4_of_4‚0.9614438414573669 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49242421984672546 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04708901792764664 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7452892661094666 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6468374133110046 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9948979616165161 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7424242496490479 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38660213351249695 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7404559850692749 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_3_of_3‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4743589758872986 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7974721789360046 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960839152336121 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4963183104991913 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4579314589500427 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.559233546257019 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.991651177406311 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8289912343025208 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16595979034900665 6"Purity/*FRt*SRqqt1FBt/event_purity‚0.7294961810112 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922013282775879 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 *"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.620791494846344 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8972249627113342 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981770515441895 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08847856521606445 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 *jet/accuracy_1_of_1‚0.3181983530521393 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 % _timestamp‚1.7848448613667538e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18076035380363464 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10272588580846786 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7247148156166077 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45170292258262634 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.84375 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9761667847633362 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5315858721733093 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_3‚0.5407988429069519 *jet/accuracy_2_of_4‚0.8005134463310242 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6952812075614929 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6851152777671814 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5892810821533203 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04858174920082092 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6416774392127991 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6904235482215881 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6373221278190613 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969591498374939 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7910423874855042 *jet/accuracy_1_of_3‚0.9187732338905334 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6956891417503357 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6018978953361511 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21068111062049866 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6619046926498413 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1883668750524521 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9980158805847168 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21066929399967194 +jet/accuracy_2_of_2‚0.14332175254821777 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16598108410835266 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6443870663642883 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6177514791488647 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.34963366389274597 epoch‚10 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18836621940135956 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38547274470329285 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6180375814437866 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_2‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19539447128772736 particle/accuracy_1_of_4‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6288504600524902 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16168269515037537 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025174930691719055 (particle/accuracy‚0.8449649214744568 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7475593686103821 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8183194398880005 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7432824373245239 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04472890496253967 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012803362682461739 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1651599109172821 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12331492453813553 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7577815651893616 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04002081975340843 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8541666865348816 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43619734048843384 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9818052649497986 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5721037983894348 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8537120223045349 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7295827865600586 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1875 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.786247730255127 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6704311966896057 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8289912343025208 _runtime‚5624.724833384 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8853000402450562 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.620231568813324 6validation_average_jet_accuracy‚0.3598572611808777 _step‚254 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6625429391860962 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9927433729171753 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19789455831050873 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7866083979606628 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05374891683459282 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21916761994361877 particle/accuracy_2_of_3‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967115521430969 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6549596190452576 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6963528990745544 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16862764954566956 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6549596190452576 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.055958930402994156 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2359752655029297 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1875 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.993336021900177 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03229544311761856 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17196889221668243 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7653846144676208 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9768880009651184 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2591298222541809 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3820382058620453 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.389506995677948 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19539447128772736 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6895986795425415 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20767274498939514 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6470404267311096 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.692684531211853 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6471472382545471 +jet/accuracy_3_of_4‚0.11347367614507675 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 *jet/accuracy_1_of_4‚0.9714088439941406 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.977791965007782 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9919642806053162 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9971705675125122 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963151812553406 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888960123062134 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17847929894924164 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10833848267793655E 0È ’¤w 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6904235482215881 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6373221278190613 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969591498374939 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7910423874855042 *jet/accuracy_1_of_3‚0.9187732338905334 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6956891417503357 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6018978953361511 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21068111062049866 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6619046926498413 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1883668750524521 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9980158805847168 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21066929399967194 +jet/accuracy_2_of_2‚0.14332175254821777 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.1659810841Ô˜Óz50835266 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6443870663642883 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6177514791488647 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.34963366389274597 epoch‚10 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18836621940135956 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38547274470329285 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6180375814437866 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_2‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19539447128772736 particle/accuracy_1_of_4‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6288504600524902 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16168269515037537 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025174930691719055 (particle/accuracy‚0.8449649214744568 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7475593686103821 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8183194398880005 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7432824373245239 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04472890496253967 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012803362682461739 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1651599109172821 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12331492453813553 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7577815651893616 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04002081975340843 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8541666865348816 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43619734048843384 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9818052649497986 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5721037983894348 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8537120223045349 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7295827865600586 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1875 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.786247730255127 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6704311966896057 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8289912343025208 _runtime‚5624.724833384 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8853000402450562 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.620231568813324 6validation_average_jet_accuracy‚0.3598572611808777 _step‚254 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6625429391860962 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9927433729171753 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19789455831050873 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7866083979606628 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05374891683459282 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21916761994361877 particle/accuracy_2_of_3‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967115521430969 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6549596190452576 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6963528990745544 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16862764954566956 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6549596190452576 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.055958930402994156 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2359752655029297 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1875 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.993336021900177 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03229544311761856 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17196889221668243 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7653846144676208 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9768880009651184 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2591298222541809 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3820382058620453 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.389506995677948 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19539447128772736 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6895986795425415 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20767274498939514 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6470404267311096 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.692684531211853 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6471472382545471 +jet/accuracy_3_of_4‚0.11347367614507675 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 *jet/accuracy_1_of_4‚0.9714088439941406 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.977791965007782 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9919642806053162 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9971705675125122 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963151812553406 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888960123062134 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17847929894924164 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10833848267793655 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9973951578140259 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04002081975340843 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30431628227233887 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1514056921005249 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7928844094276428 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03800711780786514 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5846153497695923 +particle/specificity‚0.8990258574485779 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6490334272384644 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20919108390808105 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 )loss/val_total_loss‚9.450759575968942 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5306274890899658 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 trainer/global_step‚3079 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6423673033714294 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38085299730300903 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06409718096256256 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7136649489402771 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7552627921104431 *jet/accuracy_1_of_2‚0.7682105898857117 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26840507984161377 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1948956549167633 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.886854350566864 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22679026424884796 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7424242496490479 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915882349014282 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ,jet/accuracy_3_of_3‚0.059080176055431366 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9830737709999084 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7972584962844849 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 /particle/accuracy_4_of_4‚0.9614438414573669 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49242421984672546 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04708901792764664 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7452892661094666 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6468374133110046 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9948979616165161 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7424242496490479 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38660213351249695 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7404559850692749 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_3_of_3‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4743589758872986 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7974721789360046 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960839152336121 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4963183104991913 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4579314589500427 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.559233546257019 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.991651177406311 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8289912343025208 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16595979034900665 6"Purity/*FRt*SRqqt1FBt/event_purity‚0.7294961810112 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922013282775879 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 *"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.620791494846344 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8972249627113342 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981770515441895 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08847856521606445 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 *jet/accuracy_1_of_1‚0.3181983530521393 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 % _timestamp‚1.7848448613667538e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18076035380363464 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10272588580846786 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7247148156166077 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45170292258262634 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.84375 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9761667847633362 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5315858721733093 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_3‚0.5407988429069519 *jet/accuracy_2_of_4‚0.8005134463310242 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6952812075614929 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6851152777671814 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5892810821533203 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04858174920082092 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6416774392127991þí¦ý‘“‚(:† ½¤ŠÓáˆÒ³ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚59.0 gpu.0.powerWatts‚106.461' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.487 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0— ø3;”‚(:°  ½¤ŠÓìàò³ disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 147.4296875 disk.drbd1289.in‚0* proc.memory.percent‚11.728525161743164" disk.nvme0n1.out‚ 19562.9765625 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2882.40234375 disk.drbd1817.in‚0$ memory_percent‚92.12695757548013 proc.cpu.threads‚17) disk./.usagePercent‚70.51336760403787 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 147.4296875 disk.drbd2041.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 62.74609375 disk.dm-0.out‚ 19562.9765625 disk.drbd2078.out‚0 disk.drbd2041.out‚0 network.recv‚344679 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0# disk./.usageGB‚1260.85062789917 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 26.0400390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1934.87890625 cpu‚25.30309635225983 network.sent‚ 5.470272e+06 disk.drbd1552.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 62.74609375¯.X@•‚(:µ  ̤ŠÓºÑ‚´* proc.memory.percent‚11.729876200358074 cpu‚25.30070765905143 disk.rbd1.in‚0 disk.nvme0n1.in‚ 62.74609375 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 62.74609375 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2111.in‚0! proc.memory.rssMB‚ 2882.734375 network.sent‚ 5.47038e+06 disk.drbd1938.out‚0) proc.memory.availableMB‚ 2915.33203125 network.recv‚344787) disk./.usagePercent‚70.51358008852735 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0# disk.dm-0.out‚20532.94775390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 26.0400390625$ memory_percent‚88.13748359680176 disk.drbd1550.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.8544273376465 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 148.109375 disk.drbd2041.out‚0 disk.sda2.out‚ 148.109375 disk.drbd1356.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚20532.94775390625 disk.drbd1550.out‚0Iåi{ž–‚(:“ ̤ŠÓ™èß³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚282.625' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.20833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E Œž—‚(:“ Û¤ŠÓ„†º³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚293.326' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.77533333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º-‰ A˜‚(:¶  Û¤ŠÓ¼Ôܳ disk.drbd1817.in‚0$ memory_percent‚88.13748359680176 network.sent‚ 5.470446e+06) disk./.usagePercent‚70.51357795514895 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 26.2041015625 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 62.74609375 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.sda.out‚ 148.3359375 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 148.3359375 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.729876200358074 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 network.recv‚344853 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 62.74609375 disk.drbd1828.in‚0% disk.nvme0n1.out‚20534.1611328125) proc.memory.availableMB‚ 2915.33203125% disk./.usageGB‚1260.8543891906738 disk.drbd1510.out‚0 disk.drbd1938.in‚0" disk.dm-0.out‚20534.1611328125 disk.drbd1510.in‚0 cpu‚25.30084028658794 disk.drbd2041.in‚0 disk.drbd1552.in‚0ÑWÕ2™- _wandbruntime‚5663  _runtime‚5663xCÛÎȚ ' lr-AdamW/pg1‚0.00010534607199842688 ' lr-AdamW/pg2‚0.00010534607199842688 trainer/global_step‚3099 $ _timestamp‚1.784844899867867e+09 _runtime‚5663.225110469 _step‚255¿LPÍ›Ç _runtime‚5663.225110469 _step‚255 ' lr-AdamW/pg1‚0.00010534607199842688 ' lr-AdamW/pg2‚0.00010534607199842688 trainer/global_step‚3099 $ _timestamp‚1.784844899867867e+09ÿM†ü 2œ- _wandbruntime‚5663  _runtime‚5663r¨Ûó'íO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚896110673148416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3955650219008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚256 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3236344500736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3234999064576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26925866 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3234999064576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚892874328647680 _runtime‚5663.226425519 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚892874328647680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8264520 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28786089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244348292313088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28796153 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248303983313408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248303574440960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚896110673148416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17707900 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55722019 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8264515 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26925866 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚892874302474240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26925860 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55711949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26925860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 408872448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244347924221952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 trainer/global_step‚3099 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28796153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚892521304518512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896109301538816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895750860605178 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚896109301538816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3236344500736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3229581514634 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3955691000320ô è’“žòO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚892874302474240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17708196 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚895752230644432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚40781312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚896110673148416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17707900 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55722019 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8264515 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26925866 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚892874302474240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26925860 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55711949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26925860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 408872448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244347924221952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 trainer/global_step‚3099 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28796153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚892521304518512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896109301538816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895750860605178 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚896109301538816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3236344500736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3229581514634 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3955691000320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚896110673148416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3955650219008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚256 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3236344500736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3234999064576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26925866 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3234999064576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚892874328647680 _runtime‚5663.226425519 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚892874328647680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8264520 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28786089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244348292313088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28796153 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248303983313408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248303574440960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3236021572608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26934227 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17713531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8267155 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248383450479616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244426878679040 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28805249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚892800080864320 _step‚257 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55739482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38676992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28795171 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9446663 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784844901346837e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55729398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚896032053219960 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚893153237938176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28795171 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚5664.704442057 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55739482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3230602316274 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚893153237938176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚892800106292288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚896390604954112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚896389233337344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28805249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3237367015936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚893153211764736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8267160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3956978568704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26934233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3236021572608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26934233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9446376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406768128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55739482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38676992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28795171 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9446663 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784844901346837e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55729398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚896032053219960 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚893153237938176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28795171 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚5664.704442057 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55739482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3230602316274 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚893153237938176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚892800106292288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚896390604954112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚896389233337344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28805249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3237367015936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚893153211764736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8267160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3956978568704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26934233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3236021572608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26934233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9446376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406768128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3236021572608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26934227 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17713531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8267155 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248383450479616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244426878679040 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366MT¸32¢- _wandbruntime‚5664  _runtime‚56643»s £ƒ epoch‚11 /loss/FRt1/detection_loss‚0.5759075844889227 1loss/SRqqt2/detection_loss‚0.7001429647414081 /loss/FBt1/detection_loss‚0.9465341130271554 /loss/FBt2/detection_loss‚0.4851912075395171 %loss/total_loss‚9.649498028649893 2loss/SRqqt1/assignment_loss‚0.6994951933564908 1loss/FBt2/assignment_loss‚0.10263921865035314 _runtime‚5664.740174546 0loss/FRt1/assignment_loss‚1.9806220068509357 2loss/FBt1/assignment_loss‚0.021988510291090994 1loss/SRqqt1/detection_loss‚1.0093809350139715 trainer/global_step‚3099 _step‚258 0loss/FRt2/assignment_loss‚0.8656941596980438 2loss/SRqqt2/assignment_loss‚0.5880030755790583 /loss/FRt2/detection_loss‚1.6738990594129448 % _timestamp‚1.7848449013829732e+09Tu ¤ˆ /loss/FRt1/detection_loss‚0.5759075844889227 1loss/SRqqt2/detection_loss‚0.7001429647414081 /loss/FBt1/detection_loss‚0.9465341130271554 /loss/FBt2/detection_loss‚0.4851912075395171 %loss/total_loss‚9.649498028649893 2loss/SRqqt1/assignment_loss‚0.6994951933564908 1loss/FBt2/assignment_loss‚0.10263921865035314 _runtime‚5664.740174546 0loss/FRt1/assignment_loss‚1.9806220068509357 2loss/FBt1/assignment_loss‚0.021988510291090994 1loss/SRqqt1/detection_loss‚1.0093809350139715 trainer/global_step‚3099 _step‚258 0loss/FRt2/assignment_loss‚0.8656941596980438 2loss/SRqqt2/assignment_loss‚0.5880030755790583 /loss/FRt2/detection_loss‚1.6738990594129448 % _timestamp‚1.7848449013829732e+09 epoch‚11‚,™{™“¥‚(:ˆ ꤊÓ¦üÀ³ gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚254.574' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚84.858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’æ/ß;¦‚(:°  ꤊÓ°ðã³ disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0# disk.nvme0n1.out‚21504.28515625 disk.drbd1510.out‚0$ memory_percent‚88.13775380452475 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 62.74609375 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.dm-0.out‚21504.28515625! proc.memory.rssMB‚ 2882.734375 cpu‚25.300793735028762 disk.sda.out‚ 148.8515625 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 26.2041015625 proc.cpu.threads‚17 disk.dm-0.in‚ 62.74609375 disk.drbd1289.in‚0 disk.drbd1356.out‚0 network.recv‚346597 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0' proc.memory.availableMB‚ 2915.265625 disk.sda1.out‚0 disk.sda.in‚0.03125$ disk./.usageGB‚1260.850570678711 disk.drbd1552.out‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51336440397026 disk.sda1.in‚0 disk.sda2.out‚ 148.8515625 disk.drbd2111.in‚0* proc.memory.percent‚11.729876200358074 disk.drbd1584.in‚0 disk.drbd1817.out‚0 network.sent‚ 5.530501e+06 disk.drbd1938.out‚0 disk.drbd1550.out‚0Sç‚(:’ ù¤ŠÓÎòݳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚270.08' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.02666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç <Çk¨‚(:µ  ù¤ŠÓö°þ³ disk.sda1.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 149.328125 proc.cpu.threads‚17& disk.nvme0n1.out‚21505.25634765625 disk.sda2.in‚0.03125 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 149.328125 disk.drbd1673.in‚0) disk./.usagePercent‚70.51336440397026 disk.nvme0n1.in‚ 62.74609375 disk.drbd1965.in‚0) proc.memory.availableMB‚ 2915.26171875 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0$ memory_percent‚88.13776969909668* proc.memory.percen5ÕAÕt‚11.729876200358074 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 62.74609375 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.rbd1.out‚ 26.236328125 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 network.recv‚346663 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1510.in‚0 disk.drbd1828.out‚0! proc.memory.rssMB‚ 2882.734375 network.sent‚ 5.530567e+06 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0# disk.dm-0.out‚21505.25634765625 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1552.in‚0$ disk./.usageGB‚1260.850570678711 disk.drbd1828.in‚0 cpu‚25.300702293866955ó›ñ“©‚(:ˆ ˆ¥ŠÓ³¹Ó³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚292.143' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.381 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07 äeEª‚(:º  ˆ¥ŠÓ¼×ï³ disk.drbd1584.out‚0! proc.memory.rssMB‚ 2882.734375 network.recv‚348919% disk./.usageGB‚1260.8499336242676 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 26.236328125) disk./.usagePercent‚70.51332877655084 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.sda2.out‚ 149.73046875# disk.dm-0.out‚21505.49853515625 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 149.73046875 disk.dm-0.in‚ 62.74609375 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚21505.49853515625 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0) proc.memory.availableMB‚ 2915.41015625 cpu‚25.300614315323728 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 62.74609375 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda.in‚0.03125* proc.memory.percent‚11.729876200358074 disk.drbd1356.out‚0 disk.drbd1931.in‚0 network.sent‚ 5.621286e+06 disk.drbd1817.out‚0$ memory_percent‚88.137165705362956Zw¢ž«‚(:“ —¥ŠÓªÏ³ gpu.0.gpu‚62.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.052' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.01733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J+È=¬‚(:²  —¥ŠÓÑÇî³ disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 cpu‚25.300482317824383 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.sda.out‚ 150.2109375 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚21505.77294921875 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0! proc.memory.rssMB‚ 2882.734375 disk.sda2.out‚ 150.2109375 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0# memory_percent‚88.1371815999349 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51332877655084% disk./.usageGB‚1260.8499336242676 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 26.400390625 network.recv‚348985 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0# disk.dm-0.out‚21505.77294921875 disk.nvme0n1.in‚ 63.7421875& proc.memory.availableMB‚ 2915.40625* proc.memory.percent‚11.729876200358074 disk.drbd1584.out‚0 disk.drbd1510.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 network.sent‚ 5.621352e+06 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0\_»Lž­‚(:“ ¦¥ŠÓ锲³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.211' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.07033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÙ‘B®‚(:·  ¦¥ŠÓ¶¯Ö³ disk.drbd1931.out‚0& disk.nvme0n1.out‚21506.19482421875 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8511199951172 disk.drbd2118.in‚0 proc.cpu.threads‚17 network.sent‚ 5.626928e+06 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1552.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1584.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0! proc.memory.rssMB‚ 2882.734375 disk.sda2.out‚ 150.50390625 disk.drbd1938.in‚0 network.recv‚350123 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 cpu‚25.30043943005838) disk./.usagePercent‚70.51339512461934 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1552.in‚0$ memory_percent‚88.13719749450684) proc.memory.availableMB‚ 2915.40234375 disk.drbd1817.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚21506.19482421875 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.sda.in‚0.03125 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 26.400390625 disk.sda.out‚ 150.50390625 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125ÁbŸ¯‚(:” µ¥ŠÓ³äÒ³ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚301.565' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.52166666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆM‡*;°‚(:°  µ¥ŠÓ»¶ñ³ disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 26.4326171875 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0$ memory_percent‚88.13730875651041 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 63.7421875 disk.drbd1226.in‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.sda.in‚0.03125) disk./.usagePercent‚70.51339405793013" disk.dm-0.out‚21506.7021484375 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚21506.7021484375* proc.memory.percent‚11.729876200358074 disk.sda2.out‚ 150.9296875 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0$ proc.memory.availableMB‚2915.375 network.sent‚ 5.627078e+06 disk.drbd1828.in‚0 disk.drbd2118.in‚0% disk./.usageGB‚1260.8511009216309 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1817.in‚0 network.recv‚350273 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 63.7421875 proc.cpu.threads‚17 disk.sda.out‚ 150.9296875 disk.drbd1673.out‚0 cpu‚25.300393093207294\èfé2±- _wandbruntime‚5749  _runtime‚5749„¹Ï!DzÁ _step‚259 & lr-AdamW/pg1‚0.0001049396463192812 & lr-AdamW/pg2‚0.0001049396463192812 trainer/global_step‚3149 % _timestamp‚1.7848449864320483e+09 _runtime‚5749.789321007`K6iÌ³Æ & lr-AdamW/pg1‚0.0001049396463192812 & lr-AdamW/pg2‚0.0001049396463192812 trainer/global_step‚3149 % _timestamp‚1.7848449864320483e+09 _runtime‚5749.789321007 _step‚259ƒÀqÍ62´- _wandbruntime‚5749  _runtime‚5749î…tÔó'µíO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29252105 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚906819766999040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3286188620800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 % _timestamp‚1.7848449864332235e+09 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29242041 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8396515 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248277242519552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚909742053176786 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚906819793172480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚910107327229440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚906819793172480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29252105 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4018590224896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3287534056960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9592459 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4018639394816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3149 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27344216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27344210 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252295832744448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29242041 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17988979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910105955619840 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9592116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚5749.790712558 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56586251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3286188620800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8396520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248277610610688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112¼`Zø'¶òO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27344216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27344210 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252295832744448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29242041 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17988979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910105955619840 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9592116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚5749.790712558 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56586251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3286188620800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8396520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248277610610688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29252105 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚906819766999040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3286188620800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 % _timestamp‚1.7848449864332235e+09 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29242041 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8396515 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248277242519552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚909742053176786 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚906819793172480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚910107327229440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚906819793172480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29252105 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4018590224896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3287534056960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9592459 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4018639394816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3149„ëê•i2·- _wandbruntime‚5751  _runtime‚5751«z63 ¸÷M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56613784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56603700 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚910023245791568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56603700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚907098702462976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8399160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4019891505152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚907098676289536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5751.266169458 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚906740169553120 TEDevicæÇ4ÊeStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248356196976640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3287211128832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚910387259035136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27352577 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚907098702462976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252376088481792 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚906740194981088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848449879085562e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17994512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8399155 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚348 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9595352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248355828885504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17994164 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3149 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29251123 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚907098676289536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3281706199082 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29261201 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27352583 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56613784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29251123 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9595009 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚910021875752202 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050.`÷'¹üM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3287211128832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚910387259035136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27352577 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚907098702462976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252376088481792 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚906740194981088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848449879085562e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17994512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8399155 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚348 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9595352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248355828885504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17994164 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3149 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29251123 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚907098676289536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3281706199082 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29261201 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27352583 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56613784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29251123 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9595009 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚910021875752202 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56613784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56603700 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚910023245791568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56603700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚907098702462976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8399160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4019891505152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚907098676289536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5751.266169458 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚906740169553120 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248356196976640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976…iÙ2º- _wandbruntime‚5751  _runtime‚5751ß +W»‚ 2loss/SRqqt1/assignment_loss‚0.7750962528630554 1loss/FBt2/assignment_loss‚0.11313426732866172 /loss/FRt2/detection_loss‚1.4875386160366781 1loss/SRqqt1/detection_loss‚1.0686038006529135 1loss/SRqqt2/detection_loss‚0.6925617254129043 /loss/FBt2/detection_loss‚0.4970060045861828 /loss/FRt1/assignment_loss‚2.025027257756489 2loss/SRqqt2/assignment_loss‚0.5230273664372135 2loss/FBt1/assignment_loss‚0.004277710417029638 %loss/total_loss‚9.905478866577058 trainer/global_step‚3149 % _timestamp‚1.7848449879464116e+09 /loss/FBt1/detection_loss‚1.3344578732452963 _runtime‚5751.303620273 0loss/FRt2/assignment_loss‚0.7683513337519111 /loss/FRt1/detection_loss‚0.6163966580887222 epoch‚11 _step‚262Soé. ¼‡ /loss/FRt1/assignment_loss‚2.025027257756489 2loss/SRqqt2/assignment_loss‚0.5230273664372135 2loss/FBt1/assignment_loss‚0.004277710417029638 %loss/total_loss‚9.905478866577058 trainer/global_step‚3149 % _timestamp‚1.7848449879464116e+09 /loss/FBt1/detection_loss‚1.3344578732452963 _runtime‚5751.303620273 0loss/FRt2/assignment_loss‚0.7683513337519111 /loss/FRt1/detection_loss‚0.6163966580887222 epoch‚11 _step‚262 2loss/SRqqt1/assignment_loss‚0.7750962528630554 1loss/FBt2/assignment_loss‚0.11313426732866172 /loss/FRt2/detection_loss‚1.4875386160366781 1loss/SRqqt1/detection_loss‚1.0686038006529135 1loss/SRqqt2/detection_loss‚0.6925617254129043 /loss/FBt2/detection_loss‚0.4970060045861828†¨Ñ¤îž½‚(:“ Ä¥ŠÓÈÂij gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.73' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.24333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÑf?¾‚(:´  Ä¥ŠÓœ£å³& proc.memory.availableMB‚ 2915.34375 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda.out‚ 151.46484375 disk.sda2.out‚ 151.46484375 disk.drbd1965.out‚0 disk.drbd1356.in‚0$ memory_percent‚88.13743591308594 disk.drbd1510.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1584.in‚0 disk.drbd1289.out‚0! proc.memory.rssMB‚ 2882.734375 network.sent‚ 5.627144e+06) disk./.usagePercent‚70.51339512461934% disk.nvme0n1.out‚21506.9365234375 disk.sda2.in‚0.03125 network.recv‚350339 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 cpu‚25.300349778780408 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 63.7421875" disk.dm-0.out‚21506.9365234375 disk.drbd1965.in‚0* proc.memory.percent‚11.729876200358074 disk.drbd1226.out‚0% disk./.usageGB‚1260.8511199951172 disk.drbd2111.in‚0 disk.rbd1.out‚ 26.4326171875ä8F#Ÿ¿‚(:” Ó¥ŠÓ¸Ï³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.791' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.26366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·# ?À‚(:´  Ó¥ŠÓ‚Ïë³ disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51339533795718 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 26.5966796875 disk.drbd1817.in‚0 disk.sda2.out‚ 151.92578125 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0" disk.dm-0.out‚21507.4951171875 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1226.in‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.sda1.out‚0% disk.nvme0n1.out‚21507.4951171875 disk.drbd1510.in‚0$ memory_percent‚88.13743591308594 cpu‚25.300262365561164& proc.memory.availableMB‚ 2915.34375 disk.drbd1289.in‚0 disk.sda.out‚ 151.92578125 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd2118.in‚0* proc.memory.percent‚11.729876200358074 disk.dm-0.in‚ 63.7421875 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1510.out‚0 network.sent‚ 5.632667e+06% disk./.usageGB‚1260.8511238098145 disk.sda.in‚0.03125 disk.drbd1938.in‚0 network.recv‚351444 disk.drbd1965.out‚0 disk.drbd1285.in‚0㦟ð<Á‚(:±  ⥊ÓÒÌâ³ disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0' proc.memory.availableMB‚ 1943.390625 disk.drbd1550.in‚0$ memory_percent‚92.09232330322266 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.729876200358074 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 26.5966796875 disk.drbd1285.in‚0 disk.sda2.out‚ 152.265625 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.sda1.out‚0 disk.sda.out‚ 152.265625 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.sda.in‚0.03125 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 63.7421875 cpu‚25.300391122266003 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0! proc.memory.rssMB‚ 2882.734375 network.recv‚351552 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0% disk.nvme0n1.out‚21507.6552734375 network.sent‚ 5.632775e+06 disk.drbd2118.in‚0" disk.dm-0.out‚21507.6552734375 disk.drbd1965.in‚0) disk./.usagePercent‚70.51339533795718% disk./.usageGB‚1260.8511238098145 disk.drbd1584.out‚0 disk.dm-0.in‚ 63.7421875Ÿ‘œŸÂ‚(:” ⥊ӻг gpu.0.gpu‚37.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚320.959' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚106.98633333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öSš¡žÃ‚(:“ ñ¥ŠÓ€Øس gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚257.632' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.87733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ü_>Ä‚(:³  ñ¥ŠÓˆè÷³ disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 63.7421875 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.rbd1.out‚ 26.62890625 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.09232330322266' proc.memory.availableMB‚ 1943.390625 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.729876200358074) disk./.usagePercent‚70.51326882861756 disk.sda.out‚ 152.77734375 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚21507.97607421875 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 152.77734375 disk.drbd1289.out‚0 disk.dm-0.in‚ 63.7421875# disk.dm-0.out‚21507.97607421875 disk.drbd1965.out‚0 disk.drbd1510.out‚0 network.recv‚351660 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd2041.out‚0 cpu‚25.30021733985673 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 network.sent‚ 5.632883e+06 disk.drbd2111.in‚0! proc.memory.rssMB‚ 2882.734375$ disk./.usageGB‚1260.848861694336 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1931.in‚0'q“Å‚(:ˆ €¦ŠÓʃö³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.733' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.911 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V->Æ‚(:³  €¦ŠÓô…‘´ disk.drbd1931.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.09078152974448# disk.dm-0.out‚21508.12451171875 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 153.1640625 disk.drbd1285.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 153.1640625 disk.drbd1938.in‚0 disk.drbd1356.in‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 26.72265625 disk.drbd1584.out‚0) proc.memory.availableMB‚ 1943.76953125 network.recv‚351726 disk.drbd1817.in‚0 disk.drbd1289.out‚0) disk./.usagePercent‚70.51326882861756 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 cpu‚25.30025769642987 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 network.sent‚ 5.632949e+06& disk.nvme0n1.out‚21508.12451171875 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0$ disk./.usageGB‚1260.848861694336 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 63.7421875š/«|žÇ‚(:“ ¦ŠÓ¤ùų gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚233.726' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.90866666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒTcÐ@È‚(:µ  ¦ŠÓ¸¬ä³ disk.drbd1817.in‚0 disk.nvme0n1.in‚ 63.7421875& disk.nvme0n1.out‚21508.49755859375 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 26.88671875 proc.cpu.threads‚17) disk./.usagePercent‚70.51326882861756 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0 network.recv‚352439 disk.drbd2118.out‚0 disk.sda.in‚0.03125 cpu‚25.30030279352956 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125$ memory_percent‚92.09078152974448 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 153.52734375 disk.drbd1931.in‚0 disk.drbd1965.in‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1226.in‚0 disk.dm-0.in‚ 63.7421875# disk.dm-0.out‚21508.49755859375 network.sent‚ 5.638472e+06 disk.sda.out‚ 153.52734375 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0$ disk./.usageGB‚1260.848861694336 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1943.76953125á¾(2É- _wandbruntime‚5836  _runtime‚58364GÉÊà % _timestamp‚1.7848450734458425e+09 _runtime‚5836.803074285 _step‚263 ' lr-AdamW/pg1‚0.00010452714192247031 ' lr-AdamW/pg2‚0.00010452714192247031 trainer/global_step‚3199°ÓhœÎËÈ _runtime‚5836.803074285 _step‚263 ' lr-AdamW/pg1‚0.00010452714192247031 ' lr-AdamW/pg2‚0.00010452714192247031 trainer/global_step‚3199 % _timestamp‚1.7848450734458425e+09‡}'·÷2Ì- _wandbruntime‚5836  _runtime‚5836VvB—7ÍìO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8528520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9739691 trainer/global_step‚3199 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚920765231523840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚Q¨©Ä»#27762566 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252206560817152 _runtime‚5836.804514321 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚924102609700864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚920765257697280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27762560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18268211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚920401456468144 $ _timestamp‚1.784845073446949e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27762566 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3337378177024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚920401481896112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8528515 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚924103981310464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9739330 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29708057 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3337378177024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256287577452032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29708057 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3338723613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3338723613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4081078387712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256288007296000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3331789280250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29697993 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252206928908288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚923734615787648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18267845 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4081016634880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚920765257697280 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256288007296000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3331789280250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29697993 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252206928908288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚923734615787648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18267845 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4081016634880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚920765257697280 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚924102609700864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚920765257697280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27762560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18268211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚920401456468144 $ _timestamp‚1.784845073446949e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27762566 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3337378177024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚920401481896112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8528515 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚924103981310464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9739330 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29708057 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3337378177024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256287577452032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29708057 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3338723613184ˆi6nC2Ï- _wandbruntime‚5838  _runtime‚5838Ë~ÝÅü&ÐöM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚924382541499392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27770933 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3332810081890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚5838.286966691 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3339746128384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3338400685056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252285515274240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57478002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29717153 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚921044166987776 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57488086 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8531160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252285147183104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚924013068323810 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚920680283669888 trainer/global_step‚3199 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29707075 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3338400685056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57488086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27770933 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚924014438363176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚920680258241920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚924383913116160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚363 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3339746128384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚924383913116160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29707075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9742211 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18273366 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9742574 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4082314295808 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚921044140814336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256367401830400 $ _timestamp‚1.784845074929315e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27770927 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29717153 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864¼pðQý ÑûM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57488086 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8531160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252285147183104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚924013068323810 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚920680283669888 trainer/global_step‚3199 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29707075 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3338400685056 C=DeviceStatsMonito Z•1r.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57488086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27770933 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚924014438363176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚920680258241920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚924383913116160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚363 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3339746128384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚924383913116160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29707075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9742211 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18273366 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9742574 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4082314295808 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚921044140814336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256367401830400 $ _timestamp‚1.784845074929315e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27770927 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29717153 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚924382541499392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27770933 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3332810081890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚5838.286966691 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3339746128384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3338400685056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252285515274240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57478002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29717153 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚921044166987776‰–¥2Ò- _wandbruntime‚5838  _runtime‚58385¹k9 Óƒ /loss/FBt1/detection_loss‚1.1646951409056783 _runtime‚5838.317484144 0loss/FRt1/assignment_loss‚1.9210164109182677 2loss/SRqqt1/assignment_loss‚0.6614914823836737 1loss/SRqqt1/detection_loss‚1.1156353571209099 %loss/total_loss‚9.814074950732218 0loss/FRt2/assignment_loss‚0.8718417767820688 2loss/SRqqt2/assignment_loss‚0.5312790695013677 3loss/FBt1/assignment_loss‚0.0005101556773577334 1loss/FBt2/assignment_loss‚0.07630989095250369 /loss/FRt2/detection_loss‚1.5144565520743871 /loss/FBt2/detection_loss‚0.7516469528330211 % _timestamp‚1.7848450749602528e+09 .loss/FRt1/detection_loss‚0.549696483689008 epoch‚11 trainer/global_step‚3199 _step‚266 1loss/SRqqt2/detection_loss‚0.6554956778939733Ò¶úŠÔˆ 1loss/SRqqt2/detection_loss‚0.6554956778939733 /loss/FBt1/detection_loss‚1.1646951409056783 _runtime‚5838.317484144 0loss/FRt1/assignment_loss‚1.9210164109182677 2loss/SRqqt1/assignment_loss‚0.6614914823836737 1loss/SRqqt1/detection_loss‚1.1156353571209099 %loss/total_loss‚9.814074950732218 0loss/FRt2/assignment_loss‚0.8718417767820688 2loss/SRqqt2/assignment_loss‚0.5312790695013677 3loss/FBt1/assignment_loss‚0.0005101556773577334 1loss/FBt2/assignment_loss‚0.07630989095250369 /loss/FRt2/detection_loss‚1.5144565520743871 /loss/FBt2/detection_loss‚0.7516469528330211 % _timestamp‚1.7848450749602528e+09 .loss/FRt1/detection_loss‚0.549696483689008 epoch‚11 trainer/global_step‚3199 _step‚266ŠØAPžÕ‚(:“ ž¦ŠÓÀ°®³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚269.666' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.88866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔMöûAÖ‚(:¶  ž¦ŠÓ¬Ãϳ disk.drbd1550.in‚0) proc.memory.availableMB‚ 1943.70703125! proc.memory.rssMB‚ 2882.734375 network.sent‚ 5.63858e+06 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1285.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0# disk.dm-0.out‚21508.78271484375$ memory_percent‚92.09103584289551 network.recv‚352547 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 154.01953125 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1510.out‚0% disk./.usageGB‚1260.8491134643555 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1510.in‚0 cpu‚25.300260996445704 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51328290891506 disk.rbd1.out‚ 26.88671875& disk.nvme0n1.out‚21508.78271484375 disk.drbd1817.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.729876200358074 disk.sda.out‚ 154.01953125 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0ú¼itž×‚(:“ ­¦ŠÓ£ß³ gpu.0.gpu‚85.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.138' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.71266666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"à¦%>Ø‚(:³  ­¦ŠÓÍ‘þ³ disk.drbd1938.in‚0 disk.rbd1.out‚ 27.12890625 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚21509.17822265625 disk.drbd1510.out‚0$ memory_percent‚92.09105173746744 disk.drbd1938.out‚0' proc.memory.availableMB‚ 1943.703125 disk.drbd1552.in‚0 network.recv‚352613 disk.drbd1965.in‚0 disk.sda2.out‚ 154.4140625 disk.drbd1356.out‚0 disk.drbd1828.out‚0 network.sent‚ 5.638646e+06 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 cpu‚25.300214677091663 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 154.4140625 disk.drbd1965.out‚0% disk./.usageGB‚1260.8490943908691 disk.drbd1673.in‚0 disk.dm-0.in‚ 63.7421875# disk.dm-0.out‚21509.17822265625 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125) disk./.usagePercent‚70.51328184222585 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0…'PžÙ‚(:“ ¼¦ŠÓ„¨Ä³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚274.315' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.43833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýî`w>Ú‚(:³  ¼¦ŠÓÅžã³ disk.drbd1510.out‚0* proc.memory.percent‚11.729876200358074 cpu‚25.30008916934755 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.sda1.in‚0 network.recv‚354346 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0) proc.memory.availableMB‚ 1943.69921875! proc.memory.rssMB‚ 2882.734375 proc.cpu.threads‚17 disk.sda2.in‚0.03125& disk.nvme0n1.out‚21509.37744140625 disk.drbd1552.out‚0# memory_percent‚92.0910676320394 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 154.8515625 disk.dm-0.in‚ 63.7421875# disk.dm-0.out‚21509.37744140625% disk./.usageGB‚1260.8490753173828 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.sda2.out‚ 154.8515625 disk.nvme0n1.in‚ 63.7421875 network.sent‚ 5.703834e+06 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 27.12890625) disk./.usagePercent‚70.51328077553664 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0óŽTYžÛ‚(:“ ˦ŠÓƌ۳ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.347' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.78233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,j?î5Ü‚(:ª  ˦ŠÓ•èû³ disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 network.recv‚354454 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 27.46484375 cpu‚25.300002894288617) proc.memory.availableMB‚ 1943.69921875 disk.rbd1.in‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51328184222585 disk.drbd1584.in‚0 disk.sda2.out‚ 155.109375 disk.drbd2041.in‚0 disk.drbd1550.in‚0# memory_percent‚92.0910676320394 disk.drbd2118.in‚0 disk.drbd2041.out‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 network.sent‚ 5.703942e+06% disk./.usageGB‚1260.8490943908691 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0" disk.nvme0n1.out‚ 21509.6796875 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 21509.6796875 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 155.109375 disk.drbd1965.in‚0* proc.memory.percent‚11.729876200358074 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0M¸4:Ý‚(:¯  Ú¦ŠÓßÕܳ disk.drbd1938.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2111.in‚0! proc.memory.rssMB‚ 2882.734375) disk./.usagePercent‚70.51328077553664 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.rbd1.in‚0) proc.memory.availableMB‚ 1943.94921875 cpu‚25.299962986991012 disk.sda2.out‚ 155.47265625 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 27.46484375 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.dm-0.out‚ 21510.0703125 disk.drbd1285.out‚0* proc.memory.percent‚11.729876200358074 network.sent‚ 5.704008e+06 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.09005037943521 disk.drbd1938.out‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 network.recv‚354520 disk.drbd1356.in‚0% disk./.usageGB‚1260.8490753173828 disk.sda.out‚ 155.47265625 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1828.out‚0" disk.nvme0n1.out‚ 21510.0703125Xçf“Þ‚(:ˆ Ú¦ŠÓË·¹³ gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚288.447' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.149 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌZî2ß- _wandbruntime‚5923  _runtime‚5923àÆ7£Éàà % _timestamp‚1.7848451604431648e+09 _runtime‚5923.800406158 _step‚267 ' lr-AdamW/pg1‚0.00010410861287786964 ' lr-AdamW/pg2‚0.00010410861287786964 trainer/global_step‚3249f—Æ£ÎáÈ ' lr-AdamW/pg1‚0.00010410861287786964 ' lr-AdamW/pg2‚0.00010410861287786964 trainer/global_step‚3249 % _timestamp‚1.7848451604431648e+09 _runtime‚5923.800406158 _step‚267‹ˆ‚§â2â- _wandbruntime‚5923  _runtime‚5923T+ó'ãíO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8660515 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30153945 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3388567733248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256136247205888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3384237774344 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚5923.801803209 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚938099263781888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28180910 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30153945 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58344925 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚938099263781888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚938100635391488 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28180916 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3382893163058 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256135879114752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚937724438320002 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934710722222080 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚934341570584912 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260279784059392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚937725808359256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18547479 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚934341545156944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 trainer/global_step‚3249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389913169408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30164009 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30164009 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28180910 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389913169408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18547834 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9887314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4143536853504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4143489780736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934710696048640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0**)AäòO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58334855 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚355 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚350 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚934341570584912 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260279784059392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚937725808359256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18547479 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚934341545156944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 trainer/global_step‚3249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389913169408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30164009 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30164009 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28180910 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389913169408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceSta€¹ñ<ötsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18547834 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9887314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4143536853504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4143489780736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934710696048640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8660515 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30153945 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3388567733248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256136247205888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3384237774344 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚5923.801803209 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚938099263781888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28180910 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30153945 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58344925 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚938099263781888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚938100635391488 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28180916 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3382893163058 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256135879114752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚937724438320002 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934710722222080 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58362388 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28189277 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28189283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256214465480704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4144732986368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934989605339136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58362388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9890242 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389590241280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934989631512576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389590241280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18553044 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28189277 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚938005630934784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚269 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934989605339136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260359198467072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934620372358688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3383913964698 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8663155 % _timestamp‚1.7848451619240732e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934989631512576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚938004260895418 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8663160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0jO$å'éûM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4144732986368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934989605339136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58362388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9890242 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389590241280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934989631512576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389590241280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18553044 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28189277 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚938005630934784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚269 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934989605339136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260359198467072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934620372358688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3383913964698 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8663155 % _timestamp‚1.7848451619240732e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934989631512576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚938004260895418 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8663160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28189283 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 trainer/global_step‚3249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30163027 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58362388 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28189277 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28189283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256214465480704v×.2ê- _wandbruntime‚5925  _runtime‚5925Ww˜ë /loss/FRt1/detection_loss‚0.5848914198503251 /loss/FRt2/detection_loss‚1.5449738639964732 1loss/SRqqt2/detection_loss‚0.7021219224147737 /loss/FBt2/detection_loss‚0.5967684771629547 1loss/FBt1/assignment_loss‚0.03774073582693827 1loss/FBt2/assignment_loss‚0.10538526883234345 /loss/FBt1/detection_loss‚0.9817577340804479 trainer/global_step‚3249 _runtime‚5925.316705709 _step‚270 0loss/FRt1/assignment_loss‚1.9742504061981179 0loss/FRt2/assignment_loss‚0.8537358373108477 1loss/SRqqt1/assignment_loss‚0.584345463812893 1loss/SRqqt1/detection_loss‚0.9673043079797191 %loss/total_loss‚9.483354168573083 2loss/SRqqt2/assignment_loss‚0.5500787311072495 epoch‚11 % _timestamp‚1.7848451619595275e+09u›!… ì† 1loss/FBt1/assignment_loss‚0.03774073582693827 1loss/FBt2/assignment_loss‚0.10538526883234345 /loss/FBt1/detection_loss‚0.9817577340804479 trainer/global_step‚3249 _runtime‚5925.316705709 _step‚270 0loss/FRt1/assignment_loss‚1.9742504061981179 0loss/FRt2/assignment_loss‚0.8537358373108477 1loss/SRqqt1/assignment_loss‚0.584345463812893 1loss/SRqqt1/detection_loss‚0.9673043079797191 %loss/total_loss‚9.483354168573083 2loss/SRqqt2/assignment_loss‚0.5500787311072495 epoch‚11 % _timestamp‚1.7848451619595275e+09 /loss/FRt1/detection_loss‚0.5848914198503251 /loss/FRt2/detection_loss‚1.5449738639964732 1loss/SRqqt2/detection_loss‚0.7021219224147737 /loss/FBt2/detection_loss‚0.5967684771629547Ž'ûNží‚(:“ ø¦ŠÓÖìdz gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.796' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.59866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~¢BÅAî‚(:¶  ø¦ŠÓßÆå³ disk.drbd1510.in‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0" disk.dm-0.out‚21510.9208984375 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51328205556369 disk.drbd1817.in‚0 disk.drbd1817.out‚0$ memory_percent‚92.09179878234863 disk.nvme0n1.in‚ 63.7421875! proc.memory.rssMB‚ 2882.734375 network.recv‚356402 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1943.51953125 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 156.27734375 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 network.sent‚ 5.769323e+06 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.sda.out‚ 156.27734375% disk.nvme0n1.out‚21510.9208984375% disk./.usageGB‚1260.8490982055664 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.729876200358074 cpu‚25.29979516929565 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 63.7421875 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 27.4970703125 disk.drbd1550.in‚0 disk.drbd1552.in‚0BÜé­„ï‚(:” ‡§ŠÓœšÚ³ gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚160.634' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚53.544666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0. Ù‚uncorrectedMemoryErrors‚0ó¹Í®?ð‚(:´  ‡§ŠÓýÊø³ disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0" disk.dm-0.out‚21511.7060546875 disk.rbd1.in‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1943.76953125 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.09078152974448 disk.drbd1965.out‚0 network.sent‚ 5.769389e+06 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 156.66015625 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 27.6611328125 disk.drbd1510.out‚0) disk./.usagePercent‚70.51328098887448 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0# disk./.usageGB‚1260.84907913208 disk.drbd1673.out‚0 disk.sda2.out‚ 156.66015625 disk.drbd1226.in‚0 disk.drbd1552.in‚0 cpu‚25.29975263399906 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125! proc.memory.rssMB‚ 2882.734375 disk.dm-0.in‚ 63.7421875 network.recv‚356468 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 63.7421875% disk.nvme0n1.out‚21511.7060546875œàT“ñ‚(:ˆ –§ŠÓ©Ödz gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.794' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y\¶3@ò‚(:µ  –§ŠÓ€Éè³ disk.sda.in‚0.03125 disk.drbd2078.out‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1584.in‚0# disk.dm-0.out‚21511.97216796875 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0# memory_percent‚92.0920213063558 network.recv‚357960 disk.drbd1817.in‚0 disk.sda2.out‚ 156.93359375& disk.nvme0n1.out‚21511.97216796875) disk./.usagePercent‚70.51328333559074 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.729876200358074 network.sent‚ 5.844904e+06 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0# disk./.usageGB‚1260.84912109375 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 27.6611328125 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1943.46484375 disk.dm-0.in‚ 63.7421875 disk.drbd1828.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 cpu‚25.29958777822613 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 156.93359375 disk.drbd1931.in‚0 disk.drbd2041.in‚0ì"ôÑŸó‚(:” ¥§ŠÓ³ gpu.0.gpu‚68.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.516' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.17200000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UžÛ?ô‚(:´  ¥§ŠÓ½þس network.recv‚358026 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0) disk./.usagePercent‚70.51328120221231 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 cpu‚25.299548329502183 network.sent‚ 5.84497e+06 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚21512.7001953125 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.nvme0n1.in‚ 63.7421875( proc.memory.availableMB‚ 1943.4609375 disk.drbd1965.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 157.29296875 proc.cpu.threads‚17% disk./.usageGB‚1260.8490829467773 disk.sda.out‚ 157.29296875 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125" disk.dm-0.out‚21512.7001953125 disk.drbd2078.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1510.out‚0 disk.drbd1356.out‚0$ memory_percent‚92.09203720092773* proc.memory.percent‚11.729876200358074 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 27.693359375 disk.drbd1673.out‚0 disk.drbd1965.out‚0‰Î[ “õ‚(:ˆ ´§ŠÓ¸¤§³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.818' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.606 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸xº<ö‚(:±  ´§ŠÓ³àγ disk.drbd1289.in‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 27.693359375 disk.drbd1226.in‚0 network.sent‚ 5.917834e+06 network.recv‚359541 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 cpu‚25.29942589008245 disk.drbd1931.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1356.out‚0$ memory_percent‚92.09213256835938 disk.drbd1938.out‚0 disk.drbd2041.in‚0% disk.nvme0n1.out‚21512.8134765625 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 157.62109375 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8491020202637 disk.sda1.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 157.62109375" disk.dm-0.out‚21512.8134765625 disk.drbd2111.out‚0) disk./.usagePercent‚70.51328226890152 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 63.7421875% proc.memory.availableMB‚ 1943.4375! proc.memory.rssMB‚ 2882.734375* proc.memory.percent‚11.729876200358074¥åÊ2÷- _wandbruntime‚6010  _runtime‚6010 NqÈø $ _timestamp‚1.784845247337451e+09 _runtime‚6010.694689135 _step‚271 ' lr-AdamW/pg1‚0.00010368411404504799 ' lr-AdamW/pg2‚0.00010368411404504799 trainer/global_step‚3299qfóÍùÇ trainer/global_step‚3299 $ _timestamp‚1.784845247337451e+09 _runtime‚6010.694689135 _step‚271 ' lr-AdamW/pg1‚0.00010368411404504799 ' lr-AdamW/pg2‚0.00010368411404504799'†x2ú- _wandbruntime‚6010  _runtime‚6010#“-õ'ûïO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8792520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚952095917862912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4205984359936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59209157 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18826954 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260065565503488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260065197412352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3441102725632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264271181772288 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3439757289472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8792515 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28599266 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30609897 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3439757289472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59219227 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28599260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948656186746880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚952097289472512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3433997045866 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚948656186746880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚948281633845744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3435341657152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚952095917862912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18827308 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10034439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30619961 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224ê'œú'üôO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚952097289472512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3433997045866 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚948656186746880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚948281633845744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3435341657152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚952095917862912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18827308 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10034439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30619961 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8792520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚952095917862912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4205984359936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59209157 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18826954 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260065565503488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260065197412352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3441102725632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264271181772288 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3439757289472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8792515 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28599266 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30609897 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3439757289472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59219227 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28599260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948656186746880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824|Ç,F2ý- _wandbruntime‚6012  _runtime‚6012#^èõœþøM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10037693 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3436362458904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784845248817457e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8795155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3440779797504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18832509 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30629057 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30618979 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3435017847506 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8795160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59226606 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260144151869440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28607627 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚952375849661440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260143783778304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚948560461047488 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚952377221278208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _runtime‚6012.175086197 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59236690 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28607627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264351010896896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3440779797504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59236690 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951996823506392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚951995453467026 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 trainer/global_step‚3299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4207227118592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3442125240832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948935096037376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28607633 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30618979 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30629057 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚344”·må'ÿýM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59236690 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28607627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264351010896896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3440779797504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59236690 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951996823506392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚951995453467026 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 trainer/global_step‚3299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4207227118592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3442125240832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948935096037376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28607633 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30618979 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30629057 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10037693 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3436362458904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784845248817457e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8795155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3440779797504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18832509 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30629057 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30618979 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3435017847506 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8795160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59226606 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260144151869440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28607627 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚952375849661440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260143783778304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚948560461047488 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚952377221278208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _runtime‚6012.175086197 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‘ÓÆ x2€- _wandbruntime‚6012  _runtime‚6012¿ «X€ /loss/FRt1/detection_loss‚0.6348065923091628 .loss/FRt2/detection_loss‚1.521940029925254 1loss/SRqqt1/detection_loss‚1.0796775279056416 2loss/SRqqt1/assignment_loss‚0.6657210563788983 /loss/FBt1/detection_loss‚0.8008241420345646 epoch‚11 % _timestamp‚1.7848452488537529e+09 0loss/FRt1/assignment_loss‚1.9363095709136673 %loss/total_loss‚9.392792308296698 0loss/FRt2/assignment_loss‚0.9389420552614802 2loss/SRqqt2/assignment_loss‚0.5854731427759895 /loss/FBt2/assignment_loss‚0.056927027417423 1loss/SRqqt2/detection_loss‚0.6490473499209224 /loss/FBt2/detection_loss‚0.5180038030846045 trainer/global_step‚3299 _runtime‚6012.211013701 _step‚274 2loss/FBt1/assignment_loss‚0.005120010369091281+ô¢; ‚… trainer/global_step‚3299 _runtime‚6012.211013701 _step‚274 2loss/FBt1/assignment_loss‚0.005120010369091281 /loss/FRt1/detection_loss‚0.6348065923091628 .loss/FRt2/detection_loss‚1.521940029925254 1loss/SRqqt1/detection_loss‚1.0796775279056416 2loss/SRqqt1/assignment_loss‚0.6657210563788983 /loss/FBt1/detection_loss‚0.8008241420345646 epoch‚11 % _timestamp‚1.7848452488537529e+09 0loss/FRt1/assignment_loss‚1.9363095709136673 %loss/total_loss‚9.392792308296698 0loss/FRt2/assignment_loss‚0.9389420552614802 2loss/SRqqt2/assignment_loss‚0.5854731427759895 /loss/FBt2/assignment_loss‚0.056927027417423 1loss/SRqqt2/detection_loss‚0.6490473499209224 /loss/FBt2/detection_loss‚0.5180038030846045’_‹*8ƒ‚(:­  çŠÓÑÝÚ³ disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1510.in‚0 cpu‚25.29934380983626 disk.dm-0.in‚ 63.7421875 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd2118.in‚0) proc.memory.availableMB‚ 1943.48828125 disk.drbd2078.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51328120221231 disk.drbd1550.out‚0$ memory_percent‚92.09192593892416* proc.memory.percent‚11.729876200358074 disk.drbd1828.out‚0 network.sent‚ 5.9179e+06 disk.sda.in‚0.03125 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8490829467773 disk.sda.out‚ 157.953125 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.rbd1.out‚ 27.857421875$ disk.nvme0n1.out‚21513.115234375! disk.dm-0.out‚21513.115234375 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0! proc.memory.rssMB‚ 2882.734375 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 network.recv‚359607 disk.drbd1673.out‚0 disk.sda2.out‚ 157.953125 disk.drbd1938.in‚0 disk.drbd1965.in‚03«9¥ž„‚(:“ çŠÓÏำ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.548' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.84933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[‹Ü‡ž…‚(:“ Ò§ŠÓëódz gpu.0.gpu‚17.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚321.417' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚107.13899999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó0»)=†‚(:²  Ò§ŠÓÛßé³ disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 27.857421875 disk.drbd1550.in‚0 disk.drbd1285.out‚0 network.recv‚360599 disk.nvme0n1.in‚ 63.7421875 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.sda2.out‚ 158.2265625 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1356.out‚0 cpu‚25.299344409663178 network.sent‚ 5.923487e+06 disk.drbd1510.in‚0! proc.memory.rssMB‚ 2882.734375 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0$ memory_percent‚92.09345181783041 disk.drbd1817.in‚0$ disk.nvme0n1.out‚21513.361328125 disk.drbd2118.out‚0 disk.drbd1938.in‚0! disk.dm-0.out‚21513.361328125) disk./.usagePercent‚70.51326946863108 disk.drbd1510.out‚0% disk./.usageGB‚1260.8488731384277 disk.sda.out‚ 158.2265625 disk.drbd1289.in‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1285.in‚0) proc.memory.availableMB‚ 1943.11328125 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd1552.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0­j‘‡‚(:† ᧊Óç·Æ³ gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.68' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.56 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼l/ï?ˆ‚(:´  ᧊Óß¾ç³ disk.drbd1931.out‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 158.640625 disk.sda2.in‚0.03125& disk.nvme0n1.out‚21513.70166015625 disk.drbd1510.in‚0 network.sent‚ 5.923595e+06 disk.sda.in‚0.03125 disk.sda2.out‚ 158.640625 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 63.7421875) proc.memory.availableMB‚ 1943.07421875 disk.dm-0.in‚ 63.7421875 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 27.8896484375 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1226.in‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0# memory_percent‚92.0936107635498 disk.drbd1552.out‚0 disk.drbd2041.out‚0 cpu‚25.299181658072037% disk./.usageGB‚1260.8488731384277 disk.drbd1584.in‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.729876200358074 network.recv‚360707# disk.dm-0.out‚21513.70166015625 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51326946863108 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0v:öž‰‚(:“ ð§ŠÓöž¶³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚234.083' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.02766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þñ=õ?Š‚(:´  ð§ŠÓà†Ø³* proc.memory.percent‚11.729876200358074 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 27.8896484375& disk.nvme0n1.out‚21513.88916015625 disk.drbd1550.out‚0) disk./.usagePercent‚70.51326840194187 disk.drbd1828.in‚0 disk.drbd1817.in‚0 network.sent‚ 5.986241e+06 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125' proc.memory.availableMB‚ 1942.921875 disk.drbd1584.out‚0% disk./.usageGB‚1260.8488540649414 disk.dm-0.in‚ 63.7421875 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0# disk.dm-0.out‚21513.88916015625 proc.cpu.threads‚17 disk.drbd1828.out‚0 cpu‚25.29914370450396 disk.drbd1356.in‚0$ memory_percent‚92.09423065185547 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 network.recv‚362685 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 158.8515625 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1285.out‚0 disk.sda.out‚ 158.8515625?qÀH”‹‚(:‰ ÿ§ŠÓ›Ã´³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.468' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.156 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëayBŒ‚(:·  ÿ§ŠÓàÒ³ disk.drbd1550.out‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1673.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 63.7421875 disk.drbd2078.in‚0 disk.drbd2111.in‚0# memory_percent‚92.0942465464274 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 159.23046875 disk.drbd1817.in‚0 disk.drbd1285.out‚0 network.sent‚ 5.986307e+06 disk.drbd1584.in‚0 disk.sda.out‚ 159.23046875 disk.drbd1673.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 network.recv‚362751 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0# disk.dm-0.out‚21514.16748046875) proc.memory.availableMB‚ 1942.91796875) disk./.usagePercent‚70.51326840194187 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 28.1396484375 cpu‚25.29902338277975 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚21514.16748046875 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8488540649414 disk.drbd1817.out‚0 disk.drbd1965.out‚0! proc.memory.rssMB‚ 2882.734375 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0Ö3°@‚(:µ  Ž¨ŠÓåöû³ disk.sda2.in‚0.03125 disk.rbd1.out‚ 28.1396484375 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 network.sent‚ 5.986415e+06 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 cpu‚25.298982156146714 disk.drbd1289.in‚0 disk.sda2.out‚ 159.4921875 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.729876200358074 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0# disk.dm-0.out‚21514.54248046875 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 63.7421875) disk./.usagePercent‚70.51337933761911& disk.nvme0n1.out‚21514.54248046875 disk.sda.in‚0.03125 disk.drbd1510.out‚0 disk.drbd1550.out‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 159.4921875$ memory_percent‚92.09486643473306 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1584.out‚0 network.recv‚362859 disk.drbd1226.out‚0 disk.drbd1817.in‚0' proc.memory.availableMB‚ 1942.765625 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0% disk./.usageGB‚1260.8508377075195 disk.dm-0.in‚ 63.7421875 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0n 0Ž‚(:’ Ž¨ŠÓ‚þÛ³ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.86' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.95333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÍJu2- _wandbruntime‚6097  _runtime‚6097²RóÈ trainer/global_step‚3349 $ _timestamp‚1.784845334393361e+09 _runtime‚6097.750555839 _step‚275 ' lr-AdamW/pg1‚0.00010325370106607658 ' lr-AdamW/pg2‚0.00010325370106607658î1oÁÍ‘Ç $ _timestamp‚1.784845334393361e+09 _runtime‚6097.750555839 _step‚275 ' lr-AdamW/pg1‚0.00010325370106607658 ' lr-AdamW/pg2‚0.00010325370106607658 trainer/global_step‚3349“'幓2’- _wandbruntime‚6097  _runtime‚609766L´ “ïO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10182140 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8924520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848453343943985e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263994883801088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29017616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3490946845696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60093529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚966092571943936 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4268522533376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚966092571943936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3486445539960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3485100928674 _step‚276 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29017616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 _runtime‚6097.754952929 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚962601651271680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60083459 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070¬å]~ú'”ôO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚962601651271680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60083459 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10182140 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8924520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848453343943985e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263994883801088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29017616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3490946845696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60093529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚966092571943936 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4268522533376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚966092571943936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3486445539960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3485100928674 _step‚276 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29017616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 _runtime‚6097.754952929”¼B“w2•- _wandbruntime‚6099  _runtime‚6099å½þæþ&–øM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29025977 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60100908 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29025983 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚358 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268343236831744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3487466341712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4269766664704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19111843 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3493314797056 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31085009 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962880560562176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6099.233628674 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962880560562176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚962880534388736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3493314797056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10184688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965988016078000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3486121730314 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚962500524308320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3491969353728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60110992 _step‚277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60110992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962880534388736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1×þ§ør—ýM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚358 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268343236831744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3487466341712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4269766664704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19111843 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3493314797056 PADeviceStatsMo}¼éY‘nitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31085009 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962880560562176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6099.233628674 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962880560562176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚962880534388736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3493314797056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10184688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965988016078000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3486121730314 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚962500524308320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3491969353728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60110992 _step‚277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60110992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962880534388736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29025977 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60100908 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29025983 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653•>¶ß 2˜- _wandbruntime‚6099  _runtime‚6099´á²»™‚ /loss/FRt2/assignment_loss‚0.885219556819491 2loss/SRqqt2/assignment_loss‚0.5097634833048683 1loss/SRqqt2/detection_loss‚0.7183566926818069 /loss/FBt1/detection_loss‚1.1143096560039198 0loss/FBt2/detection_loss‚0.47206416381292926 2loss/SRqqt1/assignment_loss‚0.6408468292281032 1loss/FBt2/assignment_loss‚0.07795101716971733 %loss/total_loss‚9.500732769211954 trainer/global_step‚3349 epoch‚11 0loss/FRt1/assignment_loss‚1.9738248991802787 1loss/FBt1/assignment_loss‚0.03803498421412909 /loss/FRt1/detection_loss‚0.6192725532227115 % _timestamp‚1.7848453359094775e+09 _runtime‚6099.266743613 _step‚278 /loss/FRt2/detection_loss‚1.4709741205305737 1loss/SRqqt1/detection_loss‚0.9801148130434255¹äÒ š‡ /loss/FRt2/assignment_loss‚0.885219556819491 2loss/SRqqt2/assignment_loss‚0.5097634833048683 1loss/SRqqt2/detection_loss‚0.7183566926818069 /loss/FBt1/detection_loss‚1.1143096560039198 0loss/FBt2/detection_loss‚0.47206416381292926 2loss/SRqqt1/assignment_loss‚0.6408468292281032 1loss/FBt2/assignment_loss‚0.07795101716971733 %loss/total_loss‚9.500732769211954 trainer/global_step‚3349 epoch‚11 0loss/FRt1/assignment_loss‚1.9738248991802787 1loss/FBt1/assignment_loss‚0.03803498421412909 /loss/FRt1/detection_loss‚0.6192725532227115 % _timestamp‚1.7848453359094775e+09 _runtime‚6099.266743613 _step‚278 /loss/FRt2/detection_loss‚1.4709741205305737 1loss/SRqqt1/detection_loss‚0.9801148130434255–_‰Þ›‚(:’ ¨ŠÓî•Ö³ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚303.342' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.11399999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!®—ß=œ‚(:²  ¨ŠÓê½ö³ disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.09513664245605 disk.drbd1584.out‚0 network.sent‚ 5.99193e+06 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 159.9453125) disk./.usagePercent‚70.51337784425424 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0% disk./.usageGB‚1260.8508110046387 disk.nvme0n1.in‚ 63.7421875 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 network.recv‚363954 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1942.69921875 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 159.9453125 disk.drbd1931.out‚0! proc.memory.rssMB‚ 2882.734375 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚21515.06005859375 proc.cpu.threads‚17 disk.rbd1.out‚ 28.171875 disk.drbd1931.in‚0 cpu‚25.298903572206196 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.dm-0.in‚ 63.7421875 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0& disk.nvme0n1.out‚21515.06005859375 disk.drbd1965.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.729876200358074î¼?;‚(:°  ¬¨ŠÓÅ¥Õ³ network.recv‚364062 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 160.265625 disk.rbd1.in‚0 disk.drbd1289.out‚0) disk./.usagePercent‚70.51337379083526& disk.nvme0n1.out‚21515.43896484375 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0# disk.dm-0.out‚21515.43896484375 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1965.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.729876200358074 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 1942.69921875% disk./.usageGB‚1260.8507385253906 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 160.265625 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 28.265625 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 disk.drbd1584.in‚0! proc.memory.rssMB‚ 2882.734375 network.sent‚ 5.992038e+06 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0$ memory_percent‚92.09513664245605 disk.drbd1673.in‚0 disk.drbd1673.out‚0 cpu‚25.29996475689188 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125Þ>šŸž‚(:” ¬¨ŠÓœÀ²³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.262' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.42066666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚‘ˆŸŸ‚(:’ »¨ŠÓ¿»º³ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚316.694' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚105.56466666666667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S·[; ‚(:°  »¨ŠÓªÝس disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 160.578125 disk.nvme0n1.in‚ 63.7421875% disk.nvme0n1.out‚21515.8271484375 disk.sda1.out‚0 cpu‚25.30612482804982 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1828.in‚0 network.recv‚365209 disk.drbd1938.out‚0 disk.drbd1817.in‚0) proc.memory.availableMB‚ 1941.94140625 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51337699090287 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0$ memory_percent‚92.09822018941244" disk.dm-0.out‚21515.8271484375 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 160.578125 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 network.sent‚ 6.000288e+06 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 28.4296875 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0% disk./.usageGB‚1260.8507957458496C b¢2¡- _wandbruntime‚6135  _runtime‚6135£Çõý;¢÷w 'particle/f_score‚0.6319685578346252 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21356479823589325 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29903626441955566 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992559552192688 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7898620963096619 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982571601867676 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8286435008049011 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03399188071489334 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15833914279937744 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17974576354026794 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9993678331375122 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 *jet/accuracy_1_of_4‚0.9840968251228333 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2291666716337204 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029459258541464806 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.714629054069519 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7909556031227112 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1788901388645172 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799321293830872 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7358880043029785 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8953632712364197 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.39287611842155457 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0352107398211956 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03310391679406166 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18242795765399933 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21561500430107117 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +particle/sensitivity‚0.6243061423301697 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7323692440986633 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030208831652998924 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16443899273872375 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7024379968643188 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945054650306702 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19727876782417297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6484689116477966 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38509124517440796 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7376145124435425 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 _step‚279 )loss/val_total_loss‚9.257545935545021 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9908801317214966 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8585729002952576 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6245772242546082 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6652005314826965 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6264017224311829 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6513932347297668 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7827535271644592 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988095223903656 (particle/accuracy‚0.8439825177192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19584710896015167 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6724428534507751 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7814006805419922 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469696879386902 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9980158805847168 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.731078028678894 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022134173661470413 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954063296318054 *jet/accuracy_2_of_3‚0.5221389532089233 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6808958649635315 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.979105532169342 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6847711801528931 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2623315453529358 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7885435819625854 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2291666716337204 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19727876782417297 *jet/accuracy_2_of_4‚0.8084623217582703 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6300511956214905 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16524717211723328 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.27710622549057007 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21464331448078156 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 +jet/accuracy_3_of_4‚0.09946583211421967 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10288222134113312 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9917892217636108 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5728299021720886 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6200830340385437 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6162254214286804 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3949289917945862 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6043175458908081 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042059820145368576 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08572928607463837 7validation_average_jet_accuracy‚0.35211148858070374 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972197413444519 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929828643798828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18964917957782745 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8966637253761292 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37057042121887207 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6232894062995911 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5814102292060852 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7000000476837158 _runtime‚6135.978595405 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16981470584869385 (jet/accuracy_1_of_3‚0.90987628698349 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8904257416725159 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6229363083839417 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961757659912109 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_1‚1 epoch‚11 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2083333283662796 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7893218994140625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2083333283662796 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7379928231239319 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7359240651130676 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9817602038383484 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.61930912733078 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 +particle/specificity‚0.9039847254753113 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2603829801082611 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2096499502658844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967432022094727 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007284088525921106 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.051900122314691544 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6781488656997681 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9725590348243713 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.026092488318681717 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5307692289352417 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7490434050559998 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5186370015144348 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014108968898653984 *jet/accuracy_1_of_1‚0.3089919090270996 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16719259321689606 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5603570938110352 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 trainer/global_step‚3359 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6629297137260437 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17845074832439423 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4611193537712097 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.991651177406311 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7183906435966492 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19974012672901154 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8608223795890808 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21116343140602112 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9948979616165161 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 % _timestamp‚1.7848453726204836e+09 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1169891506433487 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8229166865348816 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7168877720832825 /particle/accuracy_3_of_4‚0.9955357313156128 )jet/accuracy_1_of_2‚0.760784924030304 +jet/accuracy_2_of_2‚0.14362892508506775 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6745924949645996 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6277276277542114 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977678656578064 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 /particle/accuracy_4_of_4‚0.9594104886054993 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16248516738414764 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15212193131446838 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8608223795890808 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996392548084259 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09943635016679764 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4314107596874237 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9739843010902405 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7373276352882385 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06257453560829163 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2115168422460556 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9942314028739929 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6277276277542114 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5580269694328308 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37012049555778503 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.658183753490448 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6983863711357117 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982455372810364 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5797564387321472 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4220559298992157 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022134173661470413,ËžX£üw ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7379928231239319 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7359240651130676 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9817602038383484 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.61930912733078 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 +particle/specificity‚0.9039847254753113 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2603829801082611 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2096499502658844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967432022094727 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007284088525921106 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.051900122314691544 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6781488656997681 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9725590348243713 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.026092488318681717 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5307692289352417 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7490434050559998 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5186370015144348 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014108968898653984 *jet/accuracy_1_of_1‚0.3089919090270996 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16719259321689606 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5603570938110352 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 trainer/global_step‚3359 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6629297137260437 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17845074832439423 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4611193537712097 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.991651177406311 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7183906435966492 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19974012672901154 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8608223795890808 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21116343140602112 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9948979616165161 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 % _timestamp‚1.7848453726204836e+09 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1169891506433487 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8229166865348816 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7168877720832825 /particle/accuracy_3_of_4‚0.9955357313156128 )jet/accuracy_1_of_2‚0.760784924030304 +jet/accuracy_2_of_2‚0.14362892508506775 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6745924949645996 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6277276277542114 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977678656578064 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 /particle/accuracy_4_of_4‚0.9594104886054993 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16248516738414764 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15212193131446838 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8608223795890808 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996392548084259 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09943635016679764 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.431410-EÉnð*7596874237 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9739843010902405 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7373276352882385 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06257453560829163 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2115168422460556 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9942314028739929 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6277276277542114 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5580269694328308 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37012049555778503 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.658183753490448 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6983863711357117 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982455372810364 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5797564387321472 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4220559298992157 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022134173661470413 'particle/f_score‚0.6319685578346252 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21356479823589325 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29903626441955566 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992559552192688 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7898620963096619 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982571601867676 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8286435008049011 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03399188071489334 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15833914279937744 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17974576354026794 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9993678331375122 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 *jet/accuracy_1_of_4‚0.9840968251228333 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2291666716337204 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029459258541464806 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.714629054069519 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7909556031227112 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1788901388645172 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799321293830872 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7358880043029785 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8953632712364197 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.39287611842155457 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0352107398211956 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03310391679406166 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18242795765399933 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21561500430107117 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +particle/sensitivity‚0.6243061423301697 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7323692440986633 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030208831652998924 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16443899273872375 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7024379968643188 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945054650306702 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19727876782417297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6484689116477966 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38509124517440796 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7376145124435425 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 _step‚279 )loss/val_total_loss‚9.257545935545021 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9908801317214966 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8585729002952576 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6245772242546082 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6652005314826965 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6264017224311829 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6513932347297668 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7827535271644592 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988095223903656 (particle/accuracy‚0.8439825177192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19584710896015167 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6724428534507751 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7814006805419922 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469696879386902 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9980158805847168 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.731078028678894 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022134173661470413 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954063296318054 *jet/accuracy_2_of_3‚0.5221389532089233 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6808958649635315 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.979105532169342 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6847711801528931 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2623315453529358 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7885435819625854 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2291666716337204 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19727876782417297 *jet/accuracy_2_of_4‚0.8084623217582703 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6300511956214905 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16524717211723328 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.27710622549057007 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21464331448078156 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 +jet/accuracy_3_of_4‚0.09946583211421967 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10288222134113312 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9917892217636108 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5728299021720886 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6200830340385437 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6162254214286804 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3949289917945862 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6043175458908081 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042059820145368576 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08572928607463837 7validation_average_jet_accuracy‚0.35211148858070374 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972197413444519 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929828643798828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18964917957782745 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8966637253761292 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37057042121887207 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6232894062995911 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5814102292060852 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7000000476837158 _runtime‚6135.978595405 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16981470584869385 (jet/accuracy_1_of_3‚0.90987628698349 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8904257416725159 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6229363083839417 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961757659912109 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_1‚1 epoch‚11 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2083333283662796 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7893218994140625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2083333283662796 ("Purity/1FRt2SRqqt*FBt/event_purity‚0—Lô†š;¤‚(:°  ʨŠÓóšü³ disk.drbd2118.in‚0 network.sent‚ 6.000354e+06 disk.drbd1828.in‚0% disk.nvme0n1.out‚21515.9873046875 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0" disk.dm-0.out‚21515.9873046875 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 161.0703125 disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 161.0703125# proc.memory.rssMB‚ 2883.01171875 cpu‚25.306592726559114 network.recv‚365275 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0( disk./.usagePercent‚70.5134042981465 disk.dm-0.in‚ 63.7421875 disk.nvme0n1.in‚ 63.7421875 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 28.4296875 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0( proc.memory.availableMB‚ 1942.3828125) proc.memory.percent‚11.73100471496582 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.sda2.in‚0.03125# memory_percent‚92.0964241027832 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8512840270996 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0›…õŽž¥‚(:“ ʨŠÓӬݳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚279.103' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.03433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ík`ž¦‚(:“ Ù¨ŠÓ‚×Ó³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚279.562' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.18733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 jD³@§‚(:µ  Ù¨ŠÓ«Õó³ disk.drbd1673.out‚0 disk.sda2.out‚ 161.39453125 disk.drbd1965.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0) proc.memory.availableMB‚ 1942.31640625% disk./.usageGB‚1260.8508262634277 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 161.39453125 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1510.out‚0 network.sent‚ 6.063122e+06 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0) proc.memory.percent‚11.73100471496582 disk.drbd1510.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1285.in‚0 cpu‚25.306537044926486 disk.drbd1938.in‚0 disk.rbd1.out‚ 28.671875 disk.sda1.out‚0 network.recv‚367044# disk.dm-0.out‚21516.34814453125 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0# memory_percent‚92.0966943105062) disk./.usagePercent‚70.51337869760559 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚21516.34814453125 disk.dm-0.in‚ 63.7421875 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1550.in‚0EQvöŸ¨‚(:” 訊Óõ´È³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.613' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.870999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`RP°<©‚(:±  訊Óïªè³ disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1356.out‚0% proc.memory.availableMB‚ 1942.3125) proc.memory.percent‚11.73100471496582 disk.sda.in‚0.03125& disk.nvme0n1.out‚21516.53173828125 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 cpu‚25.306481773109624 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0# disk.dm-0.out‚21516.53173828125 disk.drbd1965.in‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51337869760559 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 28.671875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚92.09671020507812# proc.memory.rssMB‚ 2883.01171875 network.recv‚367152 disk.sda.out‚ 161.64453125 disk.drbd1673.in‚0 disk.drbd2111.in‚0 network.sent‚ 6.06323e+06 disk.drbd1510.in‚0 disk.sda2.out‚ 161.64453125 disk.drbd2078.in‚0% disk./.usageGB‚1260.8508262634277 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17¹yþžª‚(:“ ÷¨ŠÓ‚Öâ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚274.867' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.62233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿê¡7«‚(:¬  ÷¨ŠÓÛ‘‚´% proc.memory.availableMB‚ 1942.3125# proc.memory.rssMB‚ 2883.01171875 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 29.0078125 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0$ disk.nvme0n1.out‚21516.861328125 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1285.in‚0 network.recv‚367218( disk./.usagePercent‚70.5133797642948 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.nvme0n1.in‚ 63.7421875 disk.sda.out‚ 162.02734375 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 cpu‚25.30630391225564 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 63.7421875$ memory_percent‚92.09671020507812 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 network.sent‚ 6.063296e+06 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0) proc.memory.percent‚11.73100471496582$ disk./.usageGB‚1260.850845336914 disk.sda2.out‚ 162.02734375 disk.drbd1226.in‚0! disk.dm-0.out‚21516.861328125 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0(4C¥2¬- _wandbruntime‚6204  _runtime‚6204ˆi¨Ç­Á _runtime‚6204.374516513 _step‚280 ' lr-AdamW/pg1‚0.00010281743035823573 ' lr-AdamW/pg2‚0.00010281743035823573 trainer/global_step‚3399 # _timestamp‚1.78484544101726e+09Ôî1!Ì®Æ trainer/global_step‚3399 # _timestamp‚1.78484544101726e+09 _runtime‚6204.374516513 _step‚280 ' lr-AdamW/pg1‚0.00010281743035823573 ' lr-AdamW/pg2‚0.00010281743035823573˜\CÚï2¯- _wandbruntime‚6204  _runtime‚6204qïšô'°îO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267973201611776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4338482415616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61094500 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚982263996064557 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚979107792521216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3547754838528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚979107792521216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10356455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29525392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚329 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3547754838528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚978720837802480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10356779 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚982262626025303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3549100274688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31579172 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272311684027392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3541813650791 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982656892795904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19419862 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6204.375918147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚324 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚979107766347776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61104570 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3549100274688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚982656892795904 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982655521186304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3543158262077 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29525392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚978720812374512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚281 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680,_ÄU ±óO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784845441018392e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267972833520640 trainer/global_step‚3399 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚982655521186304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚979107766347776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31579172 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29525398 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267973201611776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4338482415616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61094500 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚982263996064557 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚979107792521216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3547754838528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚979107792521216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10356455 )oö¤VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29525392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚329 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3547754838528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚978720837802480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10356779 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚982262626025303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3549100274688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31579172 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272311684027392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3541813650791 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982656892795904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19419862 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6204.375918147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚324 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚979107766347776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61104570 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3549100274688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚982656892795904 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982655521186304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3543158262077 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29525392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚978720812374512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚281 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61122033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10359389 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3548777346560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19425759 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚323 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚978999639576256 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3550122789888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61122033 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29533765 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3550122789888 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339731798016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚978999614148288 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚3399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7848454424978757e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚ 6205.85550135 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3542834452431 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚979386675638272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10359707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚982542448600719 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272391098327552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268051787977728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31588268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9066047 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19425436 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9066052 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31578190 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29533759 _step‚282 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚979386701811712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61111949 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61122033 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29533765 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3550122789888 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339731798016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚978999614148288 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚3399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7848454424978757e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚ 6205.85550135 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3542834452431 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚979386675638272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10359707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚982542448600719 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272391098327552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268051787977728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31588268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9066047 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19425436 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9066052 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31578190 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29533759 _step‚282 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚979386701811712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61111949 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61122033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10359389 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3548777346560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19425759 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚323 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚978999639576256šˆ¡#2µ- _wandbruntime‚6205  _runtime‚6205˧ŒÏ¶ 1loss/FBt1/assignment_loss‚0.03322861523698301 1loss/FBt2/assignment_loss‚0.04295123960442164 0loss/FBt2/detection_loss‚0.48110419917308417 epoch‚12 trainer/global_step‚3399 _step‚283 0loss/FRt1/assignment_loss‚1.9633287030722486 1loss/SRqqt1/detection_loss‚1.1002429000141756 %loss/total_loss‚9.244791219610395 $ _timestamp‚1.784845442532518e+09 /loss/FRt1/detection_loss‚0.5379213054098156 1loss/SRqqt2/detection_loss‚0.6309459574986249 _runtime‚6205.889753717 2loss/SRqqt1/assignment_loss‚0.6484898441487524 2loss/SRqqt2/assignment_loss‚0.5714264733402675 /loss/FRt2/detection_loss‚1.6123829263072433 /loss/FBt1/detection_loss‚0.7672428307123482 /loss/FRt2/assignment_loss‚0.855526225092429µ©j? ·† 2loss/SRqqt1/assignment_loss‚0.6484898441487524 2loss/SRqqt2/assignment_loss‚0.5714264733402675 /loss/FRt2/detection_loss‚1.6123829263072433 /loss/FBt1/detection_loss‚0.7672428307123482 /loss/FRt2/assignment_loss‚0.855526225092429 1loss/FBt1/assignment_loss‚0.03322861523698301 1loss/FBt2/assignment_loss‚0.04295123960442164 0loss/FBt2/detection_loss‚0.48110419917308417 epoch‚12 trainer/global_step‚3399 _step‚283 0loss/FRt1/assignment_loss‚1.9633287030722486 1loss/SRqqt1/detection_loss‚1.1002429000141756 %loss/total_loss‚9.244791219610395 $ _timestamp‚1.784845442532518e+09 /loss/FRt1/detection_loss‚0.5379213054098156 1loss/SRqqt2/detection_loss‚0.6309459574986249 _runtime‚6205.889753717›X n韸‚(:” †©ŠÓ®Ÿ¯³ gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.746' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.24866666666665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÊvÊ1¹‚(:¦  †©ŠÓ·¡Ò³ disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda.out‚162.375 disk.drbd1356.in‚0 disk.drbd2111.out‚0) proc.memory.availableMB‚ 1942.27734375# proc.memory.rssMB‚ 2883.01171875 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚162.375 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1226.out‚0 network.recv‚369011 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 cpu‚25.306251879602783 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 network.sent‚ 6.12016e+06( disk./.usagePercent‚70.5134053648357 disk.drbd1965.in‚0 disk.rbd1.out‚ 29.0078125 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd2118.out‚0) proc.memory.percent‚11.73100471496582$ disk./.usageGB‚1260.851303100586 disk.sda.in‚0.03125 disk.drbd1285.in‚0 disk.drbd1938.in‚0! disk.dm-0.out‚21517.126953125 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0$ disk.nvme0n1.out‚21517.126953125 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0$ memory_percent‚92.09685325622559 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0ÎÆaÿŸº‚(:” •©ŠÓêÅ·³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚132.489' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚44.163000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÍ(»‚(:¸  •©ŠÓ´ÕÕ³ disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51340856490332 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 cpu‚25.30619672137018 network.sent‚ 6.120268e+06$ disk./.usageGB‚1260.851360321045 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 163.03515625 disk.drbd1931.out‚0) proc.memory.percent‚11.73100471496582 disk.dm-0.in‚ 63.7421875 disk.drbd1965.in‚0 disk.drbd2111.in‚0$ memory_percent‚92.09685325622559 disk.drbd1510.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 29.0400390625 network.recv‚369119# proc.memory.rssMB‚ 2883.01171875 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1942.27734375 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 163.03515625 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd13á_î56.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚21517.58544921875 disk.drbd1584.out‚0# disk.dm-0.out‚21517.58544921875 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.drbd1226.in‚0=Sͼ‚(:’ ¤©ŠÓ¢Ædz gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.47' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.15666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïb—”C½‚(:¸  ¤©ŠÓÜüç³ disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 63.7421875 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0( proc.memory.availableMB‚ 1942.2421875 network.recv‚370239 disk.sda1.out‚0 disk.drbd1550.in‚0# disk.dm-0.out‚21517.92724609375 disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.09699630737305 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 29.0400390625& disk.nvme0n1.out‚21517.92724609375 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 network.sent‚ 6.128605e+06) disk./.usagePercent‚70.51340963159251 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 cpu‚25.30614084680433 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 163.45703125 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.rbd1.in‚0) proc.memory.percent‚11.73100471496582% disk./.usageGB‚1260.8513793945312 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 163.45703125 disk.drbd1289.in‚0«¹6M’¾‚(:‡ ³©ŠÓÛú¼³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚311.061' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚103.687 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 "ÊI;¿‚(:°  ³©ŠÓ†”ܳ disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.dm-0.out‚21519.16796875 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 network.sent‚ 6.191333e+06% disk./.usageGB‚1260.8512916564941 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2883.01171875) disk./.usagePercent‚70.51340472482217$ memory_percent‚92.09725062052408 cpu‚25.30604746090476 disk.drbd1817.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1942.1796875 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 29.2041015625 disk.drbd1550.out‚0 network.recv‚372059 disk.drbd1584.in‚0# disk.nvme0n1.out‚21519.16796875) proc.memory.percent‚11.73100471496582 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.sda.out‚ 164.0703125 disk.sda2.out‚ 164.0703125 disk.drbd1931.out‚07c/žÀ‚(:“ ©ŠÓþ„À³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.851' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.95033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ëûJAÁ‚(:¶  ©ŠÓ‡Žå³ disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd2111.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2883.01171875) disk./.usagePercent‚70.51340579151137 disk.drbd1289.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1828.out‚0$ memory_percent‚92.09701220194498) proc.memory.availableMB‚ 1942.23828125 network.recv‚372167 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚21519.8916015625 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 29.2041015625 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda.out‚ 164.3828125 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0 cpu‚25.305913513388163 disk.sda2.out‚ 164.3828125 disk.drbd1931.in‚0 network.sent‚ 6.191441e+06 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1550.out‚0 disk.drbd1584.out‚0" disk.dm-0.out‚21519.8916015625 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.drbd2111.out‚0) proc.memory.percent‚11.73100471496582 disk.sda1.out‚0% disk./.usageGB‚1260.8513107299805 disk.drbd1550.in‚0 disk.drbd1938.out‚0œ"vžÂ‚(:“ Ñ©ŠÓ䫳 gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚279.451' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.15033333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑJÎm>Â(:³  Ñ©ŠÓï¥Å³% disk./.usageGB‚1260.8512725830078 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2078.out‚0# disk.dm-0.out‚21521.04931640625 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 cpu‚25.30586206073198 disk.drbd1828.in‚0 disk.rbd1.out‚ 29.236328125 disk.drbd1510.out‚0 proc.cpu.threads‚17 network.recv‚373104 disk.nvme0n1.in‚ 63.7421875 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 network.sent‚ 6.19443e+06 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0$ memory_percent‚92.09701220194498# proc.memory.rssMB‚ 2883.01171875 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.drbd1552.in‚0 disk.drbd1226.out‚0 disk.drbd1289.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 165.046875 disk.sda2.out‚ 165.046875) disk./.usagePercent‚70.51340365813297 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1942.23828125& disk.nvme0n1.out‚21521.04931640625 disk.drbd1584.out‚0 disk.drbd2078.in‚0‹‰ÌÑ2Ä- _wandbruntime‚6291  _runtime‚6291¨LüÉÅà ' lr-AdamW/pg1‚0.00010237535910661973 ' lr-AdamW/pg2‚0.00010237535910661973 trainer/global_step‚3449 % _timestamp‚1.7848455277676387e+09 _runtime‚6291.124925071 _step‚284§ˆÎÆÈ trainer/global_step‚3449 % _timestamp‚1.7848455277676387e+09 _runtime‚6291.124925071 _step‚284 ' lr-AdamW/pg1‚0.00010237535910661973 ' lr-AdamW/pg2‚0.00010237535910661973œÿþU›2Ç- _wandbruntime‚6291  _runtime‚6291¸–— õ'ÈïO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚3449 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29943748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚996255188636165 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚992660926491280 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚996652175267328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚996653546876928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61968802 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3598944394752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3594262144885 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61978872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚335 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10504044 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3592917533599 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29943748 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32025060 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚285 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993053230872576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3600289830912 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3600289830912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3598944394752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚996653546876928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848455277687597e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32025060 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚340 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32035124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271902519909376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4401108697600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚996253818596911 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚992660901063312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276303260515840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6291.126319629 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993053257046016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19699791 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29943742 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512yLëÏú'ÉôO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3600289830912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3598944394752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚996653546876928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848455277687597e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32025060 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚340 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32035124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271902519909376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4401108697600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚996253818596911 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚992660901063312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276303260515840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6291.126319629 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993053257046016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19699791 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29943742 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚3449 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29943748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚996255188636165 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚992660926491280 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚996652175267328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚996653546876928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61968802 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3598944394752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3594262144885 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61978872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚335 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10504044 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3592917533599 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29943748 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32025060 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚285 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993053230872576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3600289830912 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ¹Ó2Ê- _wandbruntime‚6292  _runtime‚6292Ì´Ãê ËùM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32044220 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3599966902784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29952115 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚996535011211693 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4402357179904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚996932107065856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29952115 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993332140163072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276383095364096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005¥þnk JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚992939702837088 trainer/global_step‚3449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3601312346112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993332140163072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚992939728265056 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9198047 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61996335 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚996933478682624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32044220 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4402408439808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚6292.606279571 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29952109 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 % _timestamp‚1.7848455292485788e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3601312346112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61996335 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10507038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271981106275328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚993332166336512 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32034142 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚996933478682624 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271980738184192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19705085 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚286 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚993332166336512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29952109 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3599966902784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61996335 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10507038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271981106275328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚993332166336512 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32034142 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚996933478682624 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271980738184192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19705085 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚286 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚993332166336512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29952109 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3599966902784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚996535011211693 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4402357179904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚996932107065856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29952115 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993332140163072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276383095364096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚992939702837088 trainer/global_step‚3449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3601312346112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993332140163072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚992939728265056 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9198047 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61996335 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚996933478682624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32044220 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4402408439808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚6292.606279571 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29952109 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824žö×ÓÉ2Í- _wandbruntime‚6292  _runtime‚6292C‰Ü Ά 2loss/FBt2/assignment_loss‚0.042244730463944356 /loss/FRt1/detection_loss‚0.5711348322120785 _runtime‚6292.639707058 2loss/SRqqt1/assignment_loss‚0.7230180884598993 4loss/FBt1/assignment_loss‚0.00013625546580600358 1loss/SRqqt1/detection_loss‚1.0744013061603417 /loss/FBt1/detection_loss‚1.4194008745253086 /loss/FBt2/detection_loss‚0.5834197227640983 epoch‚12 /loss/FRt1/assignment_loss‚1.977191299200058 0loss/FRt2/assignment_loss‚0.8814802917244778 2loss/SRqqt2/assignment_loss‚0.5342023644474567 /loss/FRt2/detection_loss‚1.5432833706367617 &loss/total_loss‚10.023003952330338 % _timestamp‚1.7848455292824776e+09 1loss/SRqqt2/detection_loss‚0.6730908162701073 trainer/global_step‚3449 _step‚287ÞâUÜÏ‹ /loss/FRt2/detection_loss‚1.5432833706367617 &loss/total_loss‚10.023003952330338 % _timestamp‚1.7848455292824776e+09 1loss/SRqqt2/detection_loss‚0.6730908162701073 trainer/global_step‚3449 _step‚287 2loss/FBt2/assignment_loss‚0.042244730463944356 /loss/FRt1/detection_loss‚0.5711348322120785 _runtime‚6292.639707058 2loss/SRqqt1/assignment_loss‚0.7230180884598993 4loss/FBt1/assignment_loss‚0.00013625546580600358 1loss/SRqqt1/detection_loss‚1.0744013061603417 /loss/FBt1/detection_loss‚1.4194008745253086 /loss/FBt2/detection_loss‚0.5834197227640983 epoch‚12 /loss/FRt1/assignment_loss‚1.977191299200058 0loss/FRt2/assignment_loss‚0.8814802917244778 2loss/SRqqt2/assignment_loss‚0.5342023644474567Ÿ4ð<6ŸÐ‚(:” à©ŠÓ”Ædz gpu.0.gpu‚57.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚328.914' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚109.63799999999999 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 #ŸN;Ñ‚(:°  à©ŠÓ…Êâ³ disk.drbd1673.in‚0 disk.drbd2041.in‚0 cpu‚25.30576725903106) proc.memory.availableMB‚ 1942.35546875 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.dm-0.in‚ 63.7421875# disk.dm-0.out‚21521.31884765625 disk.drbd1356.in‚0 disk.drbd1356.out‚0 network.sent‚ 6.194496e+06 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 29.236328125$ memory_percent‚92.09653536478677 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.sda2.out‚165.3125 disk.drbd1289.out‚0) disk./.usagePercent‚70.51340365813297% disk./.usageGB‚1260.8512725830078 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.out‚165.3125 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚21521.31884765625 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 63.7421875# proc.memory.rssMB‚ 2883.01171875 network.recv‚373170) proc.memory.percent‚11.73100471496582 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0–@³†žÒ‚(:“ ï©ŠÓþ÷¯³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.064' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.35466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨QDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1010648829348352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1006998695397376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1010650200957952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10651474 GADeviceStatsMonitor.on_train_batch_start/oveà)ñ£;!rsize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62853174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3650133950976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32481012 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4463353912832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1010246381207773 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19978886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275831838206976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9327412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30362092 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280294824028672 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1006601015180080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275831470115840 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1006600989752112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1010650200957952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1010245011168519 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9327407 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1006998721570816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19978519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 6378.26493523 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30362092 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848456149043398e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254/Á4i(áÿO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30362098 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3645366027693 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1010648829348352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1006998695397376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1010650200957952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10651474 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62853174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3650133950976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32481012 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4463353912832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1010246381207773 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19978886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275831838206976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9327412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30362092 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280294824028672 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1006601015180080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275831470115840 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1006600989752112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1010650200957952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1010245011168519 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9327407 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1006998721570816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19978519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 6378.26493523 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30362092 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848456149043398e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32491076 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280295251775488¡‘¨Vf2â- _wandbruntime‚6379  _runtime‚6379þ{U 'ã…N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1007277604687872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30370459 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1010930132763648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30370459 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1007277630861312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10654024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62870637 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6379.741485308 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9330052 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3646386829445 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1010928761146880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3652501902336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32500172 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1007277604687872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32490094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3652501902336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275910056481792 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1010524833743935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30370465 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4464606930944 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1006879791525888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32490094 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1010930132763648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32500172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9330047 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 trainer/global_step‚3499 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚364 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10654383 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275910424572928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1010526203783301 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 % _timestamp‚1.7848456163838882e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62870637 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62860553 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19984071 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3645042218047 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62860553 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824ÿçî`äŠN \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3652501902336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32500172 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1007277604687872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32490094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3652501902336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275910056481792 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1010524833743935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30370465 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4464606930944 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1006879791525888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32490094 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1010930132763648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32500172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9330047 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 trainer/global_step‚3499 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚364 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStat7îp*°sMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10654383 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275910424572928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1010526203783301 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 % _timestamp‚1.7848456163838882e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62870637 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62860553 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19984071 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3645042218047 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62860553 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1007277604687872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30370459 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1010930132763648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30370459 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1007277630861312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10654024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62870637 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6379.741485308 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9330052 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3646386829445 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1010928761146880¢ji<|2å- _wandbruntime‚6379  _runtime‚6379%æwæý 2loss/SRqqt2/assignment_loss‚0.6148467991394313 /loss/FBt2/detection_loss‚0.5269554203119683 _runtime‚ 6379.77581541 .loss/FRt1/assignment_loss‚2.09858573001364 1loss/FBt1/assignment_loss‚0.03503906125168963 1loss/FBt2/assignment_loss‚0.06285327607585371 .loss/FRt1/detection_loss‚0.554820233963049 /loss/FRt2/detection_loss‚1.5504978132506164 0loss/SRqqt1/detection_loss‚1.199374773267488 1loss/SRqqt2/detection_loss‚0.6087729891337882 0loss/FRt2/assignment_loss‚0.7806683601484365 2loss/SRqqt1/assignment_loss‚0.6437140893080433 /loss/FBt1/detection_loss‚0.9107995767365483 epoch‚12 trainer/global_step‚3499 % _timestamp‚1.7848456164185894e+09 _step‚291 %loss/total_loss‚9.5869281226005523ó¢qç‚ .loss/FRt1/assignment_loss‚2.09858573001364 1loss/FBt1/assignment_loss‚0.03503906125168963 1loss/FBt2/assignment_loss‚0.06285327607585371 .loss/FRt1/detection_loss‚0.554820233963049 /loss/FRt2/detection_loss‚1.5504978132506164 0loss/SRqqt1/detection_loss‚1.199374773267488 1loss/SRqqt2/detection_loss‚0.6087729891337882 0loss/FRt2/assignment_loss‚0.7806683601484365 2loss/SRqqt1/assignment_loss‚0.6437140893080433 /loss/FBt1/detection_loss‚0.9107995767365483 epoch‚12 trainer/global_step‚3499 % _timestamp‚1.7848456164185894e+09 _step‚291 %loss/total_loss‚9.586928122600552 2loss/SRqqt2/assignment_loss‚0.6148467991394313 /loss/FBt2/detection_loss‚0.5269554203119683 _runtime‚ 6379.77581541£ òžè‚(:“ ºªŠÓ­Ö³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.982' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.66066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O¶WDé‚(:¹  ºªŠÓÕŠâ³ disk.drbd1938.in‚0 disk.rbd1.out‚ 29.5966796875 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 cpu‚25.30525838068933 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1673.out‚0) proc.memory.percent‚11.73100471496582) disk./.usagePercent‚70.51354467444577 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 network.recv‚375270 disk.drbd1584.in‚0 disk.drbd2118.in‚0& disk.nvme0n1.out‚21523.10986328125 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 167.41015625% disk./.usageGB‚1260.8537940979004 disk.drbd1965.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.sda2.out‚ 167.41015625) proc.memory.availableMB‚ 1941.75390625 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚21523.10986328125 disk.drbd1226.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2883.01171875 network.sent‚ 6.203391e+06 disk.dm-0.in‚ 63.7421875 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0$ memory_percent‚92.09898312886557F˜Æ“ê‚(:ˆ ɪŠÓžß¹³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚265.587' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.529 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÌ3>ë‚(:³  ɪŠÓãžÜ³ disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0" disk.dm-0.out‚21523.3681640625 disk.drbd1817.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51354467444577 disk.drbd2041.out‚0 disk.rbd1.out‚ 29.62890625 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.sda1.out‚0% disk./.usageGB‚1260.8537940979004 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 cpu‚25.30520768254347% disk.nvme0n1.out‚21523.3681640625 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0$ memory_percent‚92.09898312886557 disk.sda.out‚ 167.7421875 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125) proc.memory.percent‚11.73100471496582 disk.drbd1938.out‚0 network.sent‚ 6.203457e+06 network.recv‚375336 disk.sda2.out‚ 167.7421875 disk.drbd1938.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1941.75390625l´ùèŸì‚(:” تŠÓ؂ɳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚113.619' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.873000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-àèn=í‚(:²  تŠÓЛê³ disk.drbd1965.in‚0 disk.drbd2111.out‚0 network.sent‚ 6.276644e+06 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0$ memory_percent‚92.09631284077963 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1942.41015625 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.sda1.in‚0 disk.drbd1817.in‚0% disk.nvme0n1.out‚21523.5126953125 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0) proc.memory.percent‚11.73100471496582) disk./.usagePercent‚70.51354595447282 disk.drbd1285.in‚0 disk.drbd1285.out‚0$ disk./.usageGB‚1260.853816986084 disk.dm-0.in‚ 63.7421875 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 network.recv‚377532 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0" disk.dm-0.out‚21523.5126953125 disk.drbd1828.out‚0 disk.rbd1.out‚ 29.72265625# proc.memory.rssMB‚ 2883.01171875 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 168.0859375 disk.sda2.out‚ 168.0859375 disk.drbd2041.out‚0 disk.drbd2111.in‚0 cpu‚25.30515595406718!í=“î‚(:ˆ 窊ÓÁ´²³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.609' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.203 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ôÅ?ï‚(:´  窊Ó•ä˳ disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1931.in‚0# proc.memory.rssMB‚ 2883.01171875) proc.memory.percent‚11.73100471496582 network.recv‚377598 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.sda.out‚ 168.40234375 disk.drbd1931.out‚0% disk./.usageGB‚1260.8538360595703 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0" disk.dm-0.out‚21523.8974609375 disk.drbd1285.out‚0 disk.sda2.out‚ 168.40234375) disk./.usagePercent‚70.51354702116203 disk.drbd1584.in‚0) proc.memory.availableMB‚ 1942.41015625 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 network.sent‚ 6.27671e+06 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 29.88671875 disk.drbd1226.out‚0$ memory_percent‚92.09631284077963% disk.nvme0n1.out‚21523.8974609375 cpu‚25.30499134353286 proc.cpu.threads‚17 disk.drbd2111.in‚0'3«‹Ÿð‚(:” öªŠÓ§üÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.319' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.10633333333334 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ÆAñ‚(:¶  öªŠÓƒˆû³ disk.drbd1817.out‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51354488778361 disk.drbd1673.out‚0 disk.rbd1.out‚ 29.88671875 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1226.in‚0 disk.drbd1226.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd2118.in‚0 network.sent‚ 6.276776e+06 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.09631284077963# proc.memory.rssMB‚ 2883.01171875% disk./.usageGB‚1260.8537979125977 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2111.in‚0 cpu‚25.304938210944464 disk.sda.out‚ 168.88671875 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 network.recv‚377664 disk.drbd1931.out‚0 disk.drbd1965.out‚0" disk.dm-0.out‚21524.3037109375 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1942.41015625 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 168.88671875 disk.drbd1965.in‚0 disk.drbd2041.out‚0% disk.nvme0n1.out‚21524.3037109375œÚ;ò‚(:°  …«ŠÓÑòų% disk./.usageGB‚1260.8537979125977 disk.dm-0.in‚ 63.7421875 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 network.sent‚ 6.342187e+06 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1942.41015625 disk.rbd1.out‚ 30.12890625 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51354488778361 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd2111.out‚0$ memory_percent‚92.09631284077963 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 169.34375 disk.drbd1673.out‚0 disk.drbd1938.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚21524.7119140625 disk.drbd1226.in‚0 disk.sda.in‚0.03125# proc.memory.rssMB‚ 2883.01171875 disk.drbd1550.out‚0" disk.dm-0.out‚21524.7119140625 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 network.recv‚379534 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 cpu‚25.304814823951347 disk.drbd1285.out‚0 disk.sda2.out‚ 169.34375 disk.drbd2041.out‚0°+œžó‚(:“ …«ŠÓ¥ï§³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚291.458' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.15266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þq W2ô- _wandbruntime‚6465  _runtime‚6465%æ×QÉõà ' lr-AdamW/pg1‚0.00010147404750643639 ' lr-AdamW/pg2‚0.00010147404750643639 trainer/global_step‚3549 % _timestamp‚1.7848457017405853e+09 _runtime‚6465.097806358 _step‚292Ô•¢ÎöÈ ' lr-AdamW/pg2‚0.00010147404750643639 trainer/global_step‚3549 % _timestamp‚1.7848457017405853e+09 _runtime‚6465.097806358 _step‚292 ' lr-AdamW/pg1‚0.00010147404750643639¤ª•2÷- _wandbruntime‚6465  _runtime‚6465Üãžú(øûO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30780448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30780442 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1020944186095616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30780448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9459407 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024646855038976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20257562 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63717406 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32936964 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9459412 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525602786816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32947028 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6465.099340471 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1024646855038976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10798150 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279760788413440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1024236203740127 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1020541078440912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32936964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3701323507200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30780442 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284286391200256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024645483429376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1020944186095616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279761156504576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10797794 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4525654053888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1020541103868880 % _timestamp‚1.7848457017416043e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32947028 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1020944159922176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1024237573779381 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3701323507200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1020944159922176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284286810558464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3696469910501 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10798150 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279760788413440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1024236203740127 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1020541078440912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32936964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3701323507200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30780442 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284286391200256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024645483429376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172¶÷9£544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1020944186095616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279761156504576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10797794 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4525654053888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1020541103868880 % _timestamp‚1.7848457017416043e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32947028 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1020944159922176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1024237573779381 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3701323507200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1020944159922176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284286810558464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3696469910501 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1020944186095616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30780448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9459407 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024646855038976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20257562 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63717406 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32936964 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9459412 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525602786816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32947028 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6465.099340471¥XîyU2ú- _wandbruntime‚6466  _runtime‚6466ä)Ê 'û…N C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63734855 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4526906307584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1021223095386112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20262799 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4526859241984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1024926786844672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63734855 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32956124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279839374779392 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63744939 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1020819905642656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9462047 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32946046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 _runtime‚6466.576251663 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1021223069212672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32956124 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63744939 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1024925415227904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1021223069212672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3702346015232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9462052 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1024517396354909 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _step‚294 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30788809 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3703691458560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1021223095386112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 trainer/global_step‚3549 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3703691458560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30788815 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1021223069212672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32956124 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63744939 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1024925415227904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1021223069212672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3702346015232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9462052 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1024517396354909 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _step‚294 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30788809 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3703691458560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1021223095386112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 trainer/global_step‚3549 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3703691458560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30788815 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20262799 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4526859241984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1024926786844672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63734855 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32956124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279839374779392 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63744939 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1020819905642656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9462047 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32946046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 _runtime‚6466.576251663¦£/O2ý- _wandbruntime‚6466  _runtime‚6466†DÉ”þ‚ epoch‚12 2loss/SRqqt2/assignment_loss‚0.5423379774613541 1loss/FBt1/assignment_loss‚0.16827840272773212 /loss/FBt2/detection_loss‚0.8285829427673639 &loss/total_loss‚10.124088256519464 0loss/FRt2/assignment_loss‚0.9009151499083989 2loss/SRqqt1/assignment_loss‚0.6656507036267316 % _timestamp‚1.7848457032547233e+09 _step‚295 0loss/FRt1/assignment_loss‚1.9842222409685413 1loss/FBt2/assignment_loss‚0.09413340079582047 .loss/FRt2/detection_loss‚1.530395770486135 1loss/SRqqt1/detection_loss‚1.0798335590843766 trainer/global_step‚3549 /loss/FRt1/detection_loss‚0.6164896493932084 1loss/SRqqt2/detection_loss‚0.6837332663893041 /loss/FBt1/detection_loss‚1.0295151929104966 _runtime‚6466.611916619Ï;üö ÿ‡ 1loss/SRqqt2/detection_loss‚0.6837332663893041 /loss/FBt1/detection_loss‚1.0295151929104966 _runtime‚6466.611916619 epoch‚12 2loss/SRqqt2/assignment_loss‚0.5423379774613541 1loss/FBt1/assignment_loss‚0.16827840272773212 /loss/FBt2/detection_loss‚0.8285829427673639 &loss/total_loss‚10.124088256519464 0loss/FRt2/assignment_loss‚0.9009151499083989 2loss/SRqqt1/assignment_loss‚0.6656507036267316 % _timestamp‚1.7848457032547233e+09 _step‚295 0loss/FRt1/assignment_loss‚1.9842222409685413 1loss/FBt2/assignment_loss‚0.09413340079582047 .loss/FRt2/detection_loss‚1.530395770486135 1loss/SRqqt1/detection_loss‚1.0798335590843766 trainer/global_step‚3549 /loss/FRt1/detection_loss‚0.6164896493932084§“µ(W€‚(:’ ”«ŠÓæØí³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.02' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.00666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø£Æˆ<‚(:±  ”«ŠÓ¸‘Ž´ disk.sda.out‚ 169.6328125 network.sent‚ 6.350434e+06 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1289.in‚0 disk.drbd1584.out‚0# memory_percent‚92.0954704284668$ disk./.usageGB‚1260.853801727295 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 30.12890625 disk.drbd1510.out‚0) proc.memory.percent‚11.73100471496582 network.recv‚380589 disk.drbd1931.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 cpu‚25.304760727518598 disk.sda2.out‚ 169.6328125 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1817.in‚0" disk.dm-0.out‚21525.0439453125 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1938.in‚0% disk.nvme0n1.out‚21525.0439453125 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0( proc.memory.availableMB‚ 1942.6171875 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.dm-0.in‚ 63.7421875 disk.drbd1356.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51354510112145 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1828.out‚0 disk.drbd1285.out‚0#"’“‚‚(:ˆ £«ŠÓÄæγ gpu.0.gpu‚30.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.249' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.083 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e±|̓‚(:©  £«ŠÓ¹“í³ disk.drbd1289.in‚0 disk.drbd1828.out‚0 network.sent‚ 6.3505e+06) disk./.usagePercent‚70.51354403443226 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0) proc.memory.percent‚11.73100471496582 disk.drbd2078.in‚0 disk.drbd1226.in‚0$ proc.memory.availableMB‚1942.375 disk.drbd1673.out‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚21525.587890625 network.recv‚380655 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚B úü$ 169.921875! disk.dm-0.out‚21525.587890625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 63.7421875 disk.rbd1.out‚ 30.29296875$ memory_percent‚92.09645589192709 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.sda.out‚ 169.921875 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1510.out‚0 cpu‚25.304790170027616 disk.drbd1356.out‚0 disk.drbd2111.in‚0% disk./.usageGB‚1260.8537826538086 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1550.in‚09HôÌž„‚(:“ ²«ŠÓ«µÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.248' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.74933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê)œÊ>…‚(:³  ²«ŠÓ¶²ò³ disk.drbd1965.in‚0 disk.sda.out‚ 170.47265625) proc.memory.availableMB‚ 1942.33984375 proc.cpu.threads‚17! disk.dm-0.out‚21525.787109375 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.sda.in‚0.03125 network.sent‚ 6.410674e+06 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0$ disk.nvme0n1.out‚21525.787109375 disk.nvme0n1.in‚ 63.7421875 disk.sda2.out‚ 170.47265625 cpu‚25.304664251148566 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 30.29296875 disk.drbd1226.out‚0 disk.drbd1510.in‚0$ disk./.usageGB‚1260.853801727295 disk.drbd2111.in‚0 disk.drbd2118.out‚0 network.recv‚382301 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 63.7421875 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0) proc.memory.percent‚11.73100471496582) disk./.usagePercent‚70.51354510112145 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.sda2.in‚0.03125$ memory_percent‚92.09659894307455# proc.memory.rssMB‚ 2883.01171875ìUTž†‚(:“ Á«ŠÓ§—á³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.83233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹íÀB‡‚(:·  Á«ŠÓü§ÿ³) disk./.usagePercent‚70.51353038081044 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0) proc.memory.percent‚11.73100471496582 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.09659894307455 disk.sda1.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2883.01171875 cpu‚25.304576537896256 disk.drbd1550.out‚0 disk.sda2.out‚ 170.89453125 disk.dm-0.in‚ 63.7421875 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.rbd1.out‚ 30.47265625 disk.drbd1584.out‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1942.33984375 network.recv‚382367 disk.nvme0n1.in‚ 63.7421875 disk.drbd2078.out‚0 network.sent‚ 6.41074e+06 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0& disk.nvme0n1.out‚21526.07470703125 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚21526.07470703125 disk.drbd1965.out‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8535385131836 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 170.89453125cŸóžˆ‚(:“ ЫŠÓ±Áà³ gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚107.252' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.75066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓkyVB‰‚(:·  ЫŠÓ½–´) proc.memory.percent‚11.73100471496582 network.sent‚ 6.410848e+06 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 30.53515625 disk.drbd2041.out‚0 network.recv‚382475 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51353038081044 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚21526.31396484375 disk.drbd2118.in‚0 cpu‚25.30445190297327 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2883.01171875% disk./.usageGB‚1260.8535385131836# disk.dm-0.out‚21526.31396484375 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1942.30859375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 63.7421875 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 171.17578125 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0$ memory_percent‚92.09672609965006 disk.sda2.out‚ 171.17578125 disk.drbd2041.in‚0 disk.drbd1285.in‚0Z2/a2Š- _wandbruntime‚6552  _runtime‚6552ç¾±7É‹Ã ' lr-AdamW/pg1‚0.00010101492529916974 ' lr-AdamW/pg2‚0.00010101492529916974 trainer/global_step‚3599 % _timestamp‚1.7848457888663514e+09 _runtime‚6552.223602254 _step‚296—-NÎŒÈ _runtime‚6552.223602254 _step‚296 ' lr-AdamW/pg1‚0.00010101492529916974 ' lr-AdamW/pg2‚0.00010101492529916974 trainer/global_step‚3599 % _timestamp‚1.7848457888663514e+09¨¡óE{2- _wandbruntime‚6552  _runtime‚6552Í:zÖ(ŽûO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3753858499584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283690474802176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283690106711040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1038228766350989 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9591407 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3752513063424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31198798 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1034889624446976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1034889624446976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33392916 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4588250360832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33392916 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3746229182023 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288278309999104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31198792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3753858499584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20536740 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20536392 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9591412 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1038642137510400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10944985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31198792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33402980 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1038227396311735 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1034481192557680 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591708 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0èݯ™(€P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283690106711040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1038228766350989 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9591407 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3752513063424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31198798 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1034889624446976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1034889624446976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33392916 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4588250360832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33392916 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3746229182023 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288278309999104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31198792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3753858499584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20536740 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20536392 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9591412 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1038642137510400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10944985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31198792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33402980 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1038227396311735 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1034481192557680 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591708 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3753858499584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283690474802176©ŽÐÖ2- _wandbruntime‚6553  _runtime‚6553Ddݪ£‘…N XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1038508588926517 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1035168559910912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3754881014784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3753535571456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31207159 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1038507218887151 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64619241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64619241 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64609157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31207165 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33412076 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9594052 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3754881014784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20541962 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31207165 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1038923440925696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283768693076992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33401998 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33412076 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283769061168128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1034759968903488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1034759994331456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4589460727296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1038922069308928 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3747249983663 _step‚298 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚6553.708436455 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0iq»'’ŠN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1038922069308928 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3747249983663 _step‚298 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚6553.708436455 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1038508588926517 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1035168559910912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3754881014784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3753535571456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31207159 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1038507218887151 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64619241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64619241 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64609157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31207165 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33412076 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9594052 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3754881014784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20541962 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31207165 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1038923440925696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283768693076992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33401998 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33412076 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283769061168128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1034759968903488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1034759994331456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4589460727296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824ªú§OÅ2“- _wandbruntime‚6553  _runtime‚6553Y 8”þ /loss/FRt1/detection_loss‚0.5385194051990402 1loss/SRqqt1/detection_loss‚1.0458272549452898 .loss/FBt1/detection_loss‚1.221644143263499 % _timestamp‚1.7848457903835793e+09 0loss/FRt1/assignment_loss‚1.9314458476624838 /loss/FRt2/assignment_loss‚0.766483829695013 2loss/FBt1/assignment_loss‚0.002991804498675355 _step‚299 -loss/FRt2/detection_loss‚1.52264563313653 1loss/SRqqt2/detection_loss‚0.6441554499168127 /loss/FBt2/detection_loss‚0.6054214252129467 trainer/global_step‚3599 _runtime‚6553.740841951 1loss/SRqqt2/assignment_loss‚0.537756023255198 1loss/FBt2/assignment_loss‚0.04978683968260601 %loss/total_loss‚9.457581579827307 epoch‚12 2loss/SRqqt1/assignment_loss‚0.59090392335921244É! •ƒ -loss/FRt2/detection_loss‚1.52264563313653 1loss/SRqqt2/detection_loss‚0.6441554499168127 /loss/FBt2/detection_loss‚0.6054214252129467 trainer/global_step‚3599 _runtime‚6553.740841951 1loss/SRqqt2/assignment_loss‚0.537756023255198 1loss/FBt2/assignment_loss‚0.04978683968260601 %loss/total_loss‚9.457581579827307 epoch‚12 2loss/SRqqt1/assignment_loss‚0.5909039233592124 /loss/FRt1/detection_loss‚0.5385194051990402 1loss/SRqqt1/detection_loss‚1.0458272549452898 .loss/FBt1/detection_loss‚1.221644143263499 % _timestamp‚1.7848457903835793e+09 0loss/FRt1/assignment_loss‚1.9314458476624838 /loss/FRt2/assignment_loss‚0.766483829695013 2loss/FBt1/assignment_loss‚0.002991804498675355 _step‚299«vü¹ž–‚(:“ ß«ŠÓ¦ýij gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.866' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.28866666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02új7—‚(:¬  ß«ŠÓÉñç³ disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.dm-0.in‚ 63.7421875) disk./.usagePercent‚70.51318178677849 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 cpu‚25.304443342896597 network.sent‚ 6.410914e+06 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1510.in‚0 disk.drbd1965.in‚0# memory_percent‚92.0978864034017) proc.memory.percent‚11.73100471496582 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 171.55078125 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.dm-0.out‚ 21527.328125! disk.nvme0n1.out‚ 21527.328125 disk.drbd1552.in‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 network.recv‚382541 disk.drbd1285.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 30.63671875 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0( proc.memory.availableMB‚ 1942.0234375% disk./.usageGB‚1260.8473052978516 disk.sda.out‚ 171.55078125 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1584.in‚0*öq”˜‚(:‰ î«ŠÓ㮵³ gpu.0.gpu‚69.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.034' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.678 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s A½>™‚(:³  î«ŠÓ‘ëÒ³ disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 171.83203125 disk.drbd1931.out‚0% disk./.usageGB‚1260.8473434448242 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 30.69921875 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51318392015689 disk.rbd1.in‚0 disk.drbd1550.out‚0 cpu‚25.304396713340463 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 63.7421875$ disk.nvme0n1.out‚21527.626953125 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0$ memory_percent‚92.09818840026855 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 proc.cpu.threads‚17 network.sent‚ 6.47115e+06 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2883.01171875) proc.memory.percent‚11.73100471496582 network.recv‚384147 disk.sda2.out‚ 171.83203125! disk.dm-0.out‚21527.626953125 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1941.94921875 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 63.7421875 disk.drbd1226.out‚0Ÿœ5“š‚(:ˆ ý«ŠÓ¡ÌÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.301' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.767 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s ‡ž@›‚(:µ  ý«ŠÓ‘¾û³) disk./.usagePercent‚70.51318072008928 disk.drbd1226.in‚0 disk.drbd1817.out‚0( proc.memory.availableMB‚ 1941.9453125 disk.sda2.out‚ 172.26953125 disk.dm-0.in‚ 63.7421875 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 172.26953125 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 network.sent‚ 6.471258e+06 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 63.7421875 network.recv‚384255 disk.drbd2118.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1673.out‚0% disk./.usageGB‚1260.8472862243652 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0% disk.nvme0n1.out‚21528.4072265625 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 cpu‚25.304270866237175" disk.dm-0.out‚21528.4072265625 disk.drbd2111.out‚0 disk.rbd1.out‚ 30.69921875 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0$ memory_percent‚92.09820429484049 proc.cpu.threads‚17Eˆm𞜂(:“ Œ¬ŠÓ®›Ô³ gpu.0.gpu‚98.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚289.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.64033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·‚=?‚(:´  Œ¬ŠÓàÔö³ disk.sda.out‚ 172.58984375 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0% disk./.usageGB‚1260.8473052978516 disk.rbd1.out‚ 30.7314453125 disk.drbd1584.in‚0" disk.dm-0.out‚21528.5244140625 disk.drbd1828.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.sda2.out‚ 172.58984375 disk.drbd1285.in‚0) disk./.usagePercent‚70.51318178677849 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0) proc.memory.percent‚11.73100471496582 disk.dm-0.in‚ 63.7421875 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 network.sent‚ 6.471324e+06 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 63.7421875% disk.nvme0n1.out‚21528.5244140625 cpu‚25.304186134831642 network.recv‚384321 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0% proc.memory.availableMB‚ 1941.9375 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.09823608398438# proc.memory.rssMB‚ 2883.01171875 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚03=ãûŸž‚(:” ›¬ŠÓàžµ³ gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.396' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.132000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯@¯&BŸ‚(:·  ›¬ŠÓª„Õ³ disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0% disk./.usageGB‚1260.8473052978516 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1552.out‚0 disk.sda.out‚ 172.8046875 disk.drbd1673.in‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚21528.81982421875 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0# memory_percent‚92.0982519785563 cpu‚25.304066003465948 disk.drbd1584.out‚0 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1941.93359375 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.rbd1.out‚ 30.8330078125 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 network.sent‚ 6.476978e+06 network.recv‚385469) disk./.usagePercent‚70.51318178677849 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 172.8046875 disk.nvme0n1.in‚ 63.7421875 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0# disk.dm-0.out‚21528.81982421875) proc.memory.percent‚11.73100471496582Þyð@ ‚(:µ  ª¬ŠÓÙ·±´ disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0% disk.nvme0n1.out‚21529.2646484375 disk.sda1.out‚0$ memory_percent‚92.09621747334799 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0 network.recv‚385535 disk.drbd1584.out‚0% disk./.usageGB‚1260.8475341796875 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1931.in‚0 disk.sda.out‚ 173.2734375# proc.memory.rssMB‚ 2883.01171875) disk./.usagePercent‚70.51319458704894 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 cpu‚25.30622709397201 disk.nvme0n1.in‚ 63.7421875 disk.drbd1938.in‚0" disk.dm-0.out‚21529.2646484375 network.sent‚ 6.477044e+06 disk.drbd1673.out‚0 disk.sda2.in‚0.03125) proc.memory.availableMB‚ 1942.43359375 disk.sda.in‚0.03125 disk.sda2.out‚ 173.2734375 disk.drbd1938.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 30.8955078125 disk.drbd1817.in‚0) proc.memory.percent‚11.73100471496582 disk.dm-0.in‚ 63.7421875 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0>¼Aìž¡‚(:“ ª¬ŠÓîðŵ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚293.991' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.99699999999999 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09@9—2¢- _wandbruntime‚6642  _runtime‚66427¡z8| £Ãw =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 trainer/global_step‚3639 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4623609483242035 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6466301083564758 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7576923370361328 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_3‚0.06444109976291656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8183354139328003 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6796571016311646 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03717896714806557 *jet/accuracy_1_of_2‚0.7860739827156067 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8682722449302673 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744293093681335 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872146844863892 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 _step‚300 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5726737380027771 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2638888955116272 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6878470182418823 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6612372994422913 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 particle/accuracy_2_of_3‚1 )loss/val_total_loss‚9.139602542741516 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2289808690547943 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33014506101608276 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6646551489830017 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5628673434257507 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7154608368873596 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19260410964488983 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2179962545633316 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9980829954147339 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45032092928886414 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7062410712242126 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9672732949256897 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04236673191189766 _runtime‚6642.663078318 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3055555522441864 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7591517567634583 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6609853506088257 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018328111618757248 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7validation_average_jet_accuracy‚0.37821170687675476 9"Purity/*¹.M.FRt*SRqqt0FBt/SRqqt_purity‚0.7133172750473022 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6199817657470703 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.618610680103302 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1970004290342331 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4255279004573822 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967516660690308 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 *jet/accuracy_1_of_3‚0.9255501627922058 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963151812553406 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8485921025276184 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19529783725738525 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9928756356239319 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.036690521985292435 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7519164085388184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971329569816589 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1176663264632225 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23737797141075134 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11390213668346405 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22640804946422577 *jet/accuracy_2_of_3‚0.5555295944213867 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7609987258911133 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5970856547355652 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026186306029558182 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976964592933655 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982143044471741 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2126101702451706 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8110145926475525 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9776983261108398 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9936904907226562 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 *jet/accuracy_2_of_4‚0.8431391716003418 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.805555522441864 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4634224474430084 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8716146349906921 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29566535353660583 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4415096342563629 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2364872246980667 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7252760529518127 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7294248342514038 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19809211790561676 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9689016342163086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 +jet/accuracy_3_of_4‚0.10990225523710251 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7411317229270935 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4557293951511383 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6681979298591614 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9947112202644348 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.760456383228302 *jet/accuracy_1_of_1‚0.3382459282875061 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8168583512306213 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39687618613243103 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.665138304233551 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922013282775879 +particle/sensitivity‚0.6186586022377014 +jet/accuracy_2_of_2‚0.15560948848724365 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7104000449180603 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7086581587791443 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7847221493721008 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8957541584968567 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24833248555660248 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8716146349906921 particle/accuracy_1_of_4‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29535412788391113 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7562170028686523 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3055555522441864 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5612127184867859 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23235610127449036 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3972455561161041 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6476885080337524 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.746353268623352 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6307691931724548 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 /particle/accuracy_3_of_4‚0.9913265109062195 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04307025671005249 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7475319504737854 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05037980526685715 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9113944172859192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6663008332252502 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23737797141075134 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2107303887605667 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957238435745239 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7803803086280823 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972804188728333 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18322840332984924 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7734281420707703 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.760417103767395 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 .particle/accuracy_4_of_4‚0.951833188533783 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9980829954147339 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6152133941650391 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03704208880662918 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6605876684188843 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03621102496981621 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6692222356796265 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.88326495885849 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07402224093675613 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13225547969341278 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2499949038028717 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09957684576511383 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2615811228752136 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9775030016899109 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6692222356796265 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7730674147605896 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2638888955116272 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2289276272058487 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026186306029558182 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8769024014472961 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 &particle/f_score‚0.636788547039032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9889131784439087 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18849246203899384 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04324420914053917 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 $ _timestamp‚1.784845879304943e+09 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7581750154495239 *jet/accuracy_1_of_4‚0.9796324968338013 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6574689149856567 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1922128051519394 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6309217810630798 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 particle/accuracy_1_of_2‚1 epoch‚12 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913774728775024 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7801666259765625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25097784399986267 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21046893298625946 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367×G¡cÎ;¤Èw /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9775030016899109 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6692222356796265 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7730674147605896 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2638888955116272 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2289276272058487 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026186306029558182 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8769024014472961 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 &particle/f_score‚0.636788547039032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9889131784439087 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18849246203899384 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04324420914053917 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 $ _timestamp‚1.784845879304943e+09 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7581750154495239 *jet/accuracy_1_of_4‚0.9796324968338013 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6574689149856567 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1922128051519394 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6309217810630798 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 particle/accuracy_1_of_2‚1 epoch‚12 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913774728775024 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7801666259765625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25097784399986267 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21046893298625946 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 trainer/global_step‚3639 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4623609483242035 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6466301083564758 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7576923370361328 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_3‚0.06444109976291656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8183354139328003 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6796571016311646 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03717896714806557 *jet/accuracy_1_of_2‚0.7860739827156067 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8682722449302673 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744293093681335 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872146844863892 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 _step‚300 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5726737380027771 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2638888955116272 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6878470182418823 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6612372994422913 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 particle/accuracy_2_of_3‚1 )loss/val_total_loss‚9.139602542741516 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2289808690547943 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33014506101608276 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6646551489830017 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5628673434257507 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7154608368873596 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19260410964488983 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2179962545633316 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9980829954147339 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45032092928886414 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7062410712242126 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9672732949256897 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04236673191189766 _runtime‚6642.663078318 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3055555522441864 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7591517567634583 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6609853506088257 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018328111618757248 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7validation_average_jet_accuracy‚0.37821170687675476 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7133172750473022 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6199817657470703 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.618610680103302 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1970004290342331 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4255279004573822 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967516660690308 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 *jet/accuracy_1_of_3‚0.9255501627922058 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963151812553406 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8485921025276184 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19529783725738525 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9928756356239319 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.036690521985292435 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7519164085388184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971329569816589 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1176663264632225 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23737797141075134 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11390213668346405 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22640804946422577 *jet/accuracy_2_of_3‚0.5555295944213867 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7609987258911133 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5970856547355652 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026186306029558182 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976964592933655 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982143044471741 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2126101702451706 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8110145926475525 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9776983261108398 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9936904907226562 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 *jet/accuracy_2_of_4‚0.8431391716003418 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.805555522441864 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4634224474430084 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8716146349906921 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29566535353660583 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4415096342563629 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2364872246980667 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7252760529518127 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7294248342514038 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19809211790561676 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9689016342163086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 +jet/accuracy_3_of_4‚0.10990225523710251 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7411317229270935 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4557293951511383 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6681979298591614 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9947112202644348 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.760456383228302 *jet/accuracy_1_of_1‚0.3382459282875061 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8168583512306213 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39687618613243103 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.665138304233551 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922013282775879 +particle/sensitivity‚0.6186586022377014 +jet/accuracy_2_of_2‚0.15560948848724365 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7104000449180603 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7086581587791443 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7847221493721008 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8957541584968567 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24833248555660248 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8716146349906921 particle/accuracy_1_of_4‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29535412788391113 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7562170028686523 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3055555522441864 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5612127184867859 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23235610127449036 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3972455561161041 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6476885080337524 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.746353268623352 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6307691931724548 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 /particle/accuracy_3_of_4‚0.9913265109062195 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04307025671005249 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7475319504737854 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05037980526685715 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9113944172859192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6663008332252502 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23737797141075134 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2107303887605667 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957238435745239 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7803803086280823 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972804188728333 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18322840332984924 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7734281420707703 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.760417103767395 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 .particle/accuracy_4_of_4‚0.951833188533783 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9980829954147339 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6152133941650391 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03704208880662918 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6605876684188843 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03621102496981621 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6692222356796265 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.88326495885849 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07402224093675613 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13225547969341278 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2499949038028717 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09957684576511383 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2615811228752136 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0¬ é,„‘¥‚(:† ¹¬ŠÓÓßÁ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚58.0 gpu.0.powerWatts‚105.867' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.289 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡¨ic>¦‚(:³  ¹¬ŠÓ Ýä³& disk.nvme0n1.out‚21530.12060546875 network.sent‚ 6.477152e+06 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 173.7265625 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1226.out‚0% disk./.usageGB‚1262.8024101257324 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚21530.12060546875 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 173.7265625 disk.drbd2111.in‚0) disk./.usagePercent‚70.62252148363258 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 cpu‚25.311084125316277 network.recv‚385643 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.rbd1.out‚ 30.8955078125 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 63.7421875 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 63.7421875$ memory_percent‚99.98871485392252 disk.drbd1510.out‚0 disk.drbd1285.in‚0% proc.memory.availableMB‚ 2.7734375* proc.memory.percent‚15.676053365071615 disk.drbd1938.out‚0! proc.memory.rssMB‚ 3852.546875 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚07¤,Ó“§‚(:ˆ ȬŠÓ‡à©³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚278.301' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.767 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢QKÈ9¨‚(:®  ȬŠÓàγ disk.sda2.out‚ 174.11328125 disk.drbd1356.in‚0 disk.rbd1.out‚ 30.927734375 disk.drbd1931.in‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 network.sent‚ 6.477431e+06 disk.drbd1289.out‚0 disk.dm-0.in‚64.1875 disk.drbd1965.in‚0 disk.drbd2078.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd2118.out‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51342221852512 disk.drbd1285.in‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2041.out‚0& proc.memory.availableMB‚ 2917.90625 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 174.11328125 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 network.recv‚386490 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0$ memory_percent‚88.12700907389322 cpu‚25.309155899362214 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚22499.92138671875# proc.memory.rssMB‚ 2883.01171875# disk./.usageGB‚1260.85160446167 disk.nvme0n1.in‚64.1875 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚22499.92138671875 disk.sda1.out‚0|2Õ¸2©- _wandbruntime‚6663  _runtime‚6663ŒÓÜPǪÁ & lr-AdamW/pg1‚0.0001005502388152438 & lr-AdamW/pg2‚0.0001005502388152438 trainer/global_step‚3649 % _timestamp‚1.7848459004219973e+09 _runtime‚6663.779232123 _step‚301GéaÁÌ«Æ trainer/global_step‚3649 % _timestamp‚1.7848459004219973e+09 _runtime‚6663.779232123 _step‚301 & lr-AdamW/pg1‚0.0001005502388152438 & lr-AdamW/pg2‚0.0001005502388152438­à«~g2¬- _wandbruntime‚6663  _runtime‚6663AŒß­úO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65612824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33896180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3810666494976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809321058816 NFDeviceStatsMonitor.on_train_b-—¾ú#atch_start/inactive_split.small_pool.peak‚623 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33906244 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚3649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65602754 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9730299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11118086 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1054783198873825 $ _timestamp‚1.784845900423055e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3810666494976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33906244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6663.783927491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292325818455552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4657393933824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1051395765696512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287668792612864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1050980282397648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1055206458364928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3802941904145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1055205086755328 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1051395791869952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31706574 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1054784568913079 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚8247Çc(®ÿO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402580992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4657428423680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33896180 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1055205086755328 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1051395791869952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31706574 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1054784568913079 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65612824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33896180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3810666494976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809321058816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33906244 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚3649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65602754 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9730299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11118086 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1054783198873825 $ _timestamp‚1.784845900423055e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3810666494976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33906244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6663.783927491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292325818455552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4657393933824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1051395765696512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287668792612864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1050980282397648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1055206458364928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3802941904145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65630287 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3810343566848 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1055485018553856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803962705785 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31714941 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11120773 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33905262 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31714941 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9732939 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4658702587392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1051259058743456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31714947 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31714947 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292405713475072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287747378978816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33915340 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287747010887680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20853712 % _timestamp‚1.7848459019024081e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1051674674987008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚303 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3810343566848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9732944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4658737070080 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6665.260054008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65630287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1055064391488607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805307317183 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚3649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084(ÍÈ*¡ ±ŠN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1051674701160448 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1051674674987008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65630287 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3810343566848 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1055485018553856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803962705785 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31714941 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11120773 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33905262 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31714941 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9732939 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4658702587392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1051259058743456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31714947 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31É¿º2o714947 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292405713475072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287747378978816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33915340 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287747010887680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20853712 % _timestamp‚1.7848459019024081e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1051674674987008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚303 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3810343566848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9732944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4658737070080 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6665.260054008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65630287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1055064391488607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805307317183 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚3649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672¯‹ÌT¦2²- _wandbruntime‚6665  _runtime‚6665F*Ãt³€ 0loss/FRt1/assignment_loss‚1.9286233326456426 /loss/FRt1/detection_loss‚0.5656355022134442 /loss/FRt2/detection_loss‚1.5244405203968028 /loss/FBt2/detection_loss‚0.6073750545177176 epoch‚13 _step‚304 2loss/SRqqt2/assignment_loss‚0.5325746330828398 1loss/SRqqt1/detection_loss‚0.9986342378594179 1loss/FBt1/assignment_loss‚0.08009481976589469 1loss/FBt2/assignment_loss‚0.06195995164671837 %loss/total_loss‚9.800348455099346 trainer/global_step‚3649 $ _timestamp‚1.784845901934496e+09 _runtime‚6665.291688199 0loss/FRt2/assignment_loss‚0.8655932733497904 1loss/SRqqt1/assignment_loss‚0.676872656081187 1loss/SRqqt2/detection_loss‚0.6280449429097059 /loss/FBt1/detection_loss‚1.3304995306301861IsË ´… 1loss/FBt1/assignment_loss‚0.08009481976589469 1loss/FBt2/assignment_loss‚0.06195995164671837 %loss/total_loss‚9.800348455099346 trainer/global_step‚3649 $ _timestamp‚1.784845901934496e+09 _runtime‚6665.291688199 0loss/FRt2/assignment_loss‚0.8655932733497904 1loss/SRqqt1/assignment_loss‚0.676872656081187 1loss/SRqqt2/detection_loss‚0.6280449429097059 /loss/FBt1/detection_loss‚1.3304995306301861 0loss/FRt1/assignment_loss‚1.9286233326456426 /loss/FRt1/detection_loss‚0.5656355022134442 /loss/FRt2/detection_loss‚1.5244405203968028 /loss/FBt2/detection_loss‚0.6073750545177176 epoch‚13 _step‚304 2loss/SRqqt2/assignment_loss‚0.5325746330828398 1loss/SRqqt1/detection_loss‚0.9986342378594179°jNñ…žµ‚(:“ ׬ŠÓöÁdz gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚259.024' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.34133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Õ¹;¶‚(:°  ׬ŠÓí¢æ³ disk.nvme0n1.in‚64.1875& disk.nvme0n1.out‚22500.51416015625 disk.drbd1552.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0) proc.memory.percent‚11.73100471496582 network.sent‚ 6.482875e+06& proc.memory.availableMB‚ 2917.84375 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 174.58203125$ memory_percent‚88.12726338704428 disk.drbd1931.in‚0 disk.dm-0.in‚64.1875 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 cpu‚25.309059403550247 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 network.recv‚386831% disk./.usageGB‚1260.8516235351562 disk.drbd1226.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51342328521433 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.sda.in‚0.03125# proc.memory.rssMB‚ 2883.01171875 disk.sda2.in‚0.03125# disk.dm-0.out‚22500.51416015625 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 31.115234375 disk.sda2.out‚ 174.58203125 disk.drbd2078.out‚0©¬Œ·‚(:’ 權ÓºÖ¿³ gpu.0.gpu‚10.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚59.0 gpu.0.powerWatts‚107.503' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.83433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zë<¸‚(:±  權Ó¶¼â³ disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1356.out‚0% disk./.usageGB‚1260.8477210998535 disk.drbd1673.out‚0 disk.drbd1965.out‚0 cpu‚25.30900224610297 disk.drbd1931.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.08211898803711 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.sda.out‚ 175.19140625 disk.drbd1510.in‚0 network.sent‚ 6.558626e+06 network.recv‚388670 disk.rbd1.out‚ 31.177734375 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.nvme0n1.in‚64.1875 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0( proc.memory.availableMB‚ 1945.8984375 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd1817.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51320504060314 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 175.19140625# disk.dm-0.out‚23470.20556640625 disk.sda2.in‚0.03125 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2883.01171875 disk.dm-0.in‚64.1875 disk.drbd1285.in‚0& disk.nvme0n1.out‚23470.20556640625 disk.drbd2118.in‚0Eþ†Ÿ¹‚(:” õ¬ŠÓ¡ô»³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.568' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.52266666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P•¶­9º‚(:®  õ¬ŠÓÙܳ disk.drbd1931.in‚0 disk.dm-0.in‚64.1875 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 network.recv‚388778 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 175.44140625 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0% disk./.usageGB‚1260.8477020263672 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚64.1875 disk.drbd2118.in‚0 disk.drbd2041.out‚0 network.sent‚ 6.558734e+06 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 31.177734375 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2883.01171875$ disk.nvme0n1.out‚23470.576171875 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 175.44140625 disk.sda2.in‚0.03125 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0) disk./.usagePercent‚70.51320397391395 disk.drbd1226.out‚0( proc.memory.availableMB‚ 1945.8984375) proc.memory.percent‚11.73100471496582 disk.drbd1510.out‚0! disk.dm-0.out‚23470.576171875 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.08211898803711 cpu‚25.308871297250576 disk.drbd1285.out‚0«çÚŸ»‚(:” „­ŠÓþÏȳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚127.303' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚42.434333333333335 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x³|3¼‚(:¨  „­ŠÓöÀë³ disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0! disk.dm-0.out‚23470.740234375 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.nvme0n1.in‚64.1875 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 31.2099609375 network.sent‚ 6.6443e+06 network.recv‚390550 disk.sda.out‚ 175.671875 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚23470.740234375 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1356.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd1226.out‚0 disk.sda2.out‚ 175.671875 disk.drbd1828.out‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51320397391395% disk./.usageGB‚1260.8477020263672 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0( proc.memory.availableMB‚ 1945.8984375 disk.drbd1226.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.08211898803711# proc.memory.rssMB‚ 2883.01171875 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚64.1875 disk.drbd1817.in‚0 disk.drbd2078.out‚0 cpu‚25.30881361587662,çŠË:½‚(:¯  “­ŠÓ»œá³ disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.dm-0.in‚64.1875 disk.rbd1.out‚ 31.4052734375 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0$ memory_percent‚92.08211898803711 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0) proc.memory.percent‚11.73100471496582) disk./.usagePercent‚70.51321890756279 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17 network.recv‚390658 network.sent‚ 6.644408e+06 disk.drbd2111.out‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 1945.8984375% disk./.usageGB‚1260.8479690551758 disk.drbd1817.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚23471.0751953125# proc.memory.rssMB‚ 2883.01171875 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 disk.drbd1552.in‚0 cpu‚25.308646991950834 disk.sda2.out‚ 175.9921875 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.sda.out‚ 175.9921875" disk.dm-0.out‚23471.0751953125 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚64.1875 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0†”쇞¾‚(:“ “­ŠÓ°ô¿³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.797' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.26566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëblž¿‚(:“ ¢­ŠÓö‘¼³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.625' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.54166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i}7§:À‚(:¯  ¢­ŠÓýÆß³% disk./.usageGB‚1260.8479309082031 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 network.recv‚390766 disk.drbd1817.out‚0 disk.sda2.out‚ 176.3984375 disk.drbd1289.out‚0 disk.dm-0.in‚64.1875 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 network.sent‚ 6.644516e+06 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 31.4677734375 disk.drbd1673.out‚0 disk.drbd2078.in‚0 cpu‚25.308591096820013 disk.drbd1584.in‚0 disk.sda.out‚ 176.3984375 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0$ memory_percent‚92.08313624064127( proc.memory.availableMB‚ 1945.6484375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚64.1875" disk.dm-0.out‚23471.2900390625 disk.drbd2041.in‚0 disk.sda2.in‚0.03125 disk.drbd2118.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51321677418439% disk.nvme0n1.out‚23471.2900390625 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0) proc.memory.percent‚11.73100471496582IY)Œ2Á- _wandbruntime‚6750  _runtime‚6750e³áÚÈ ' lr-AdamW/pg1‚0.00010008004896441007 ' lr-AdamW/pg2‚0.00010008004896441007 trainer/global_step‚3699 $ _timestamp‚1.784845987604966e+09 _runtime‚6750.962217443 _step‚305þlhýÍÃÇ _step‚305 ' lr-AdamW/pg1‚0.00010008004896441007 ' lr-AdamW/pg2‚0.00010008004896441007 trainer/global_step‚3699 $ _timestamp‚1.784845987604966e+09 _runtime‚6750.962217443±ÕÀ‚S2Ä- _wandbruntime‚6750  _runtime‚6750åF#2(ÅûO S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1069201740836352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296318957257216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855390398239 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66477056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚3699 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1065341230221312 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1064920345658480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296318539996160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1069203112445952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32124924 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9862304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚306 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9862299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34362196 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34352132 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1065341230221312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11266195 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291597742819328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 % _timestamp‚1.7848459876060607e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34362196 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1064920371086448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4720846346752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34352132 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3860510615040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11266526 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3860510615040 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4720846346752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34352132 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3860510615040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11266526 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3860510615040 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1069201740836352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296318957257216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855390398239 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66477056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚3699 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1065341230221312 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1064920345658480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296318539996160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1069203112445952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32124924 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9862304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚306 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9862299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34362196 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34352132 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1065341230221312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11266195 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291597742819328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 % _timestamp‚1.7848459876060607e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34362196 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181² Û=2Ç- _wandbruntime‚6752  _runtime‚6752á.x 'È…N G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66504589 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32133291 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66494505 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21134428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1069055584060215 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3855066588593 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1065620139511808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296398379534336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 trainer/global_step‚3699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1065620165685248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34361214 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32133297 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21134095 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 % _timestamp‚1.7848459890877175e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1069481672634880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66504589 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9864939 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291676697276416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34371292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1065620139511808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1065199147432256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1069483044251648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861533123072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862878566400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11269484 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32133291 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862878566400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11269156 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1065620165685248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4722101608960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32133297 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ŸÆÂ'ÉŠN TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862878566400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11269156 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1065620165685248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4722101608960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32133297 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66504589 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32133291 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66494505 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21134428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1069055584060215 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3855066588593 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1065620139511808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296398379534336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 trainer/global_step‚3699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1065620165685248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34361214 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32133297 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21134095 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 % _timestamp‚1.7848459890877175e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1069481672634880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66504589 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9864939 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291676697276416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34371292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1065620139511808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1065199147432256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1069483044251648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861533123072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862878566400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11269484 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32133291 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0³ƒT—C2Ê- _wandbruntime‚6752  _runtime‚6752š=¬®Ëû /loss/FRt1/assignment_loss‚1.944029274505839 2loss/SRqqt2/assignment_loss‚0.5790751691869428 2loss/FBt1/assignment_loss‚0.002438368689353004 /loss/FRt1/detection_loss‚0.5626562949347457 .loss/FRt2/detection_loss‚1.652354191141853 #loss/total_loss‚9.5052513042331 $ _timestamp‚1.784845989121637e+09 0loss/FRt2/assignment_loss‚0.8754984819611837 0loss/SRqqt2/detection_loss‚0.642353145435186 -loss/FBt1/detection_loss‚1.01668001394029 /loss/FBt2/detection_loss‚0.5334684452016294 epoch‚13 trainer/global_step‚3699 _runtime‚6752.478877485 2loss/SRqqt1/assignment_loss‚0.6719365777688792 1loss/FBt2/assignment_loss‚0.03346805587645226 1loss/SRqqt1/detection_loss‚0.9912932855907463 _step‚308*ŠBÿÌ€ /loss/FRt1/assignment_loss‚1.944029274505839 2loss/SRqqt2/assignment_loss‚0.5790751691869428 2loss/FBt1/assignment_loss‚0.002438368689353004 /loss/FRt1/detection_loss‚0.5626562949347457 .loss/FRt2/detection_loss‚1.652354191141853 #loss/total_loss‚9.5052513042331 $ _timestamp‚1.784845989121637e+09 0loss/FRt2/assignment_loss‚0.8754984819611837 0loss/SRqqt2/detection_loss‚0.642353145435186 -loss/FBt1/detection_loss‚1.01668001394029 /loss/FBt2/detection_loss‚0.5334684452016294 epoch‚13 trainer/global_step‚3699 _runtime‚6752.478877485 2loss/SRqqt1/assignment_loss‚0.6719365777688792 1loss/FBt2/assignment_loss‚0.03346805587645226 1loss/SRqqt1/detection_loss‚0.9912932855907463 _step‚308´…oæàŸÍ‚(:” ±­ŠÓûó˳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚304.774' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.59133333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u?Ÿl>΂(:³  ±­ŠÓî’ì³ disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1584.out‚0 network.recv‚391752% disk./.usageGB‚1260.8479499816895 disk.drbd2041.in‚0 disk.drbd1828.in‚0) proc.memory.percent‚11.73100471496582 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0( disk./.usagePercent‚70.5132178408736 disk.drbd1285.in‚0 cpu‚25.308534635494897 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 31.6474609375$ memory_percent‚92.08493232727051& disk.nvme0n1.out‚23471.61962890625 disk.drbd1938.out‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚64.1875 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 176.77734375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0# disk.dm-0.out‚23471.61962890625 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 176.77734375 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.sda.in‚0.03125 network.sent‚ 6.650084e+06 disk.drbd1552.in‚0 disk.dm-0.in‚64.1875 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1945.20703125 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0O?-†ÿÏ‚(:’ À­ŠÓË«º³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.56' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.S>$ž0.powerPercent‚91.18666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì:Ñ?Ђ(:´  À­ŠÓº¸Û³ disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.nvme0n1.in‚64.1875) proc.memory.availableMB‚ 1945.12109375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1817.out‚0 disk.drbd1226.in‚0) proc.memory.percent‚11.73100471496582 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.dm-0.in‚64.1875 disk.drbd2041.in‚0& disk.nvme0n1.out‚23471.81494140625# proc.memory.rssMB‚ 2883.01171875 disk.drbd1552.out‚0) disk./.usagePercent‚70.51321677418439 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 network.sent‚ 6.715339e+06 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.sda.out‚ 177.00390625 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.sda2.out‚ 177.00390625 network.recv‚393452 disk.drbd1550.out‚0# disk.dm-0.out‚23471.81494140625 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 cpu‚25.308406774841416$ memory_percent‚92.08528200785318 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2078.in‚0% disk./.usageGB‚1260.8479309082031 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 31.7099609375 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0ˆ EdœÑ‚(:‘ Ï­ŠÓûþó gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.3' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.76666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OY/6Ò‚(:«  Ï­ŠÓîÃä³$ disk.nvme0n1.out‚23472.267578125 disk.drbd1226.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 177.56640625 disk.drbd2041.out‚0) proc.memory.percent‚11.73100471496582 disk.sda1.out‚0 disk.drbd1289.in‚0# memory_percent‚92.0854409535726 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 cpu‚25.30846100255406 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1938.in‚0$ disk./.usageGB‚1260.847988128662 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.dm-0.in‚64.1875 disk.nvme0n1.in‚64.1875 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 1945.08203125 network.sent‚ 6.718396e+06 network.recv‚394473 disk.rbd1.out‚ 31.8115234375 disk.drbd1817.in‚0! disk.dm-0.out‚23472.267578125' disk./.usagePercent‚70.513219974252 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.sda.out‚ 177.56640625 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2883.01171875(<\žÓ‚(:“ Þ­ŠÓÉÆÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚281.876' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.95866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù5«B9Ô‚(:®  Þ­ŠÓß—‚´ disk.drbd2041.in‚0 network.recv‚395239 disk.nvme0n1.in‚64.1875 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2883.01171875% disk./.usageGB‚1260.8480072021484 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0) proc.memory.percent‚11.73100471496582) proc.memory.availableMB‚ 1945.32421875 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚23472.736328125 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1226.out‚0# memory_percent‚92.0844554901123 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0! disk.dm-0.out‚23472.736328125 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 31.8740234375 disk.drbd1510.out‚0 cpu‚25.308367605968286 network.sent‚ 6.721383e+06 disk.sda2.out‚ 177.86328125 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0( disk./.usagePercent‚70.5132210409412 disk.sda.out‚ 177.86328125 disk.dm-0.in‚64.1875 disk.drbd1356.out‚0­Ä0žÕ‚(:“ í­ŠÓÙÚ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚274.972' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.65733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ƒ„7Ö‚(:¬  í­ŠÓ“Ôÿ³ disk.drbd2111.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚64.1875$ memory_percent‚92.08587010701498 network.recv‚395389 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1828.out‚0 disk.drbd2041.in‚0! disk.dm-0.out‚23473.419921875 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚64.1875 disk.drbd2078.out‚0% disk./.usageGB‚1260.8485145568848 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 network.sent‚ 6.721533e+06 disk.drbd1584.in‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚23473.419921875 disk.drbd1510.in‚0) disk./.usagePercent‚70.51324941487404 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1938.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0( proc.memory.availableMB‚ 1944.9765625# proc.memory.rssMB‚ 2883.01171875 disk.sda1.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 31.8740234375 disk.drbd1552.out‚0 cpu‚25.30824007555106 proc.cpu.threads‚17 disk.sda.out‚ 178.3984375 disk.sda2.in‚0.03125 disk.sda2.out‚ 178.3984375 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0) proc.memory.percent‚11.73100471496582Ñ¥%‰2×- _wandbruntime‚6837  _runtime‚6837m¹5>ÆØÀ _step‚309 & lr-AdamW/pg1‚9.960441737778544e-05 & lr-AdamW/pg2‚9.960441737778544e-05 trainer/global_step‚3749 $ _timestamp‚1.784846074381672e+09 _runtime‚6837.739899706Ý,7ûËÙÅ & lr-AdamW/pg1‚9.960441737778544e-05 & lr-AdamW/pg2‚9.960441737778544e-05 trainer/global_step‚3749 $ _timestamp‚1.784846074381672e+09 _runtime‚6837.739899706 _step‚309µSQi÷2Ú- _wandbruntime‚6837  _runtime‚6837­(ÛûO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34818148 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21406649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3911700171264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11412350 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783419493888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1078860434347280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9994304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1079286694746112 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1083198394917376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11412710 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67351358 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1078860459775248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34808084 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1079286694746112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21407014 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295527061116928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _runtime‚6837.740010482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3913045607424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1083198394917376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1079286720919552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1083199766526976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3913045607424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67361428 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848460743824973e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9994299 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1082765584017041 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34818148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295527429208064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32543280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1083199766526976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1079286720919552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34808084 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911700171264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚310 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1082766954056295Yp‚(Ü€P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3911700171264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11412350 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783419493888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1078860434347280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9994304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1079286694746112 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1083198394917376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11412710 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67351358 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1078860459775248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34808084 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1079286694746112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21407014 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295527061116928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _runtime‚6837.740010482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3913045607424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1083198394917376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1079286720919552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1083199766526976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3913045607424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67361428 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848460743824973e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9994299 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1082765584017041 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34818148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295527429208064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32543280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1083199766526976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1079286720919552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34808084 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911700171264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚310 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1082766954056295 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34818148 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21406649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755¶ÈöC2Ý- _wandbruntime‚6839  _runtime‚6839du˜Þ…N QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9996939 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32551641 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300390295404032 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1083046776631823 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67368807 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1083479698332672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1079565604036608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 434031104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4784713861120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3912722679296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67368807 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34827244 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295606015574016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool/$Bó.freed‚ 4784647921152 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300390729435136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3906170471401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3912722679296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1079139261549024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32551647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67378891 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848460758601625e+09 _step‚311 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67378891 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1079565630210048 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34817166 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚3749 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1079565604036608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1083479698332672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1079565630210048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3914068122624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9996944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21412174 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11415602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1083045406592457 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181)âS*'ߊN XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1079565604036608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1083479698332672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1079565630210048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3914068122624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9996944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21412174 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11415602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1083045406592457 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9996939 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32551641 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300390295404032 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1083046776631823 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67368807 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1083479698332672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1079565604036608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 434031104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4784713861120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3912722679296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67368807 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34827244 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295606015574016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4784647921152 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300390729435136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3906170471401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3912722679296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1079139261549024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32551647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67378891 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848460758601625e+09 _step‚311 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67378891 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1079565630210048 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34817166 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚3749·Œ`¾2à- _wandbruntime‚6839  _runtime‚6839p”Êçá‚ 2loss/SRqqt2/assignment_loss‚0.5768278858334294 /loss/FBt1/detection_loss‚1.0264285787601362 % _timestamp‚1.7848460758951144e+09 0loss/FRt1/assignment_loss‚1.9157700723139506 2loss/SRqqt1/assignment_loss‚0.6745116958923482 /loss/FBt2/detection_loss‚0.5655630724240385 trainer/global_step‚3749 _runtime‚6839.252308526 0loss/FRt2/assignment_loss‚0.8097294399497705 0loss/SRqqt1/detection_loss‚1.026373717888201 1loss/SRqqt2/detection_loss‚0.6464744417475488 epoch‚13 2loss/FBt1/assignment_loss‚0.003728418243032262 2loss/FBt2/assignment_loss‚0.045135258912892696 /loss/FRt1/detection_loss‚0.5572172486297555 .loss/FRt2/detection_loss‚1.515983792420246 %loss/total_loss‚9.363743623015349 _step‚312Máî ⇠0loss/FRt1/assignment_loss‚1.9157700723139506 2loss/SRqqt1/assignment_loss‚0.6745116958923482 /loss/FBt2/detection_loss‚0.5655630724240385 trainer/global_step‚3749 _runtime‚6839.252308526 0loss/FRt2/assignment_loss‚0.8097294399497705 0loss/SRqqt1/detection_loss‚1.026373717888201 1loss/SRqqt2/detection_loss‚0.6464744417475488 epoch‚13 2loss/FBt1/assignment_loss‚0.003728418243032262 2loss/FBt2/assignment_loss‚0.045135258912892696 /loss/FRt1/detection_loss‚0.5572172486297555 .loss/FRt2/detection_loss‚1.515983792420246 %loss/total_loss‚9.363743623015349 _step‚312 2loss/SRqqt2/assignment_loss‚0.5768278858334294 /loss/FBt1/detection_loss‚1.0264285787601362 % _timestamp‚1.7848460758951144e+09¸­n«žã‚(:“ ü­ŠÓ€—³ gpu.0.gpu‚65.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.739' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.57966666666665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X·=7ä‚(:¬  ü­ŠÓ¶°â³ disk.drbd1938.out‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8503952026367 disk.sda1.out‚0) proc.memory.percent‚11.73100471496582 disk.drbd2111.out‚0 disk.drbd1673.out‚0 network.sent‚ 6.721599e+06 network.recv‚395455) disk./.usagePercent‚70.51335459042957 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.sda.out‚ 178.91015625 disk.nvme0n1.in‚64.1875# disk.dm-0.out‚23474.01025390625 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0$ proc.memory.availableMB‚1944.625# proc.memory.rssMB‚ 2883.01171875 cpu‚25.308187495191248 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 178.91015625 disk.drbd1285.out‚0& disk.nvme0n1.out‚23474.01025390625 disk.sda2.in‚0.03125 disk.drbd1226.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0$ memory_percent‚92.08730061848959 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 32.2109375 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.dm-0.in‚64.1875 disk.drbd1550.in‚0Š’žå‚(:“ ‹®ŠÓ»¡»³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.473' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.15766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·2÷7æ‚(:¬  ‹®ŠÓ¾šÝ³ disk.drbd2111.out‚0 disk.drbd1828.in‚0% disk./.usageGB‚1260.8503761291504 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 179.2109375 disk.drbd1965.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 32.3046875$ memory_percent‚92.08536148071289) proc.memory.percent‚11.73100471496582) disk./.usagePercent‚70.51335352374038 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 cpu‚25.308061085104704 network.sent‚ 6.724663e+06 disk.drbd2118.out‚0 disk.drbd1673.out‚0" disk.dm-0.out‚23475.4423828125 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2883.01171875 network.recv‚396279 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0% disk.nvme0n1.out‚23475.4423828125 disk.drbd1226.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 1945.1015625 disk.drbd1552.out‚0 disk.sda.out‚ 179.2109375 disk.drbd2041.in‚0 disk.dm-0.in‚64.1875 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚64.1875 disk.drbd1931.out‚0&M©ë6ç‚(:«  š®ŠÓ¼ÿ׳ disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda.out‚ 179.65625 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚64.1875 disk.drbd1510.out‚0 network.recv‚396345 disk.drbd2118.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51335352374038% disk./.usageGB‚1260.8503761291504 disk.dm-0.in‚64.1875 disk.drbd1931.out‚0$ memory_percent‚92.08636283874512) proc.memory.availableMB‚ 1944.85546875 network.sent‚ 6.724729e+06 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1356.out‚0) proc.memory.percent‚11.73100471496582 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 179.65625 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 32.3671875& disk.nvme0n1.out‚23476.01708984375 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 cpu‚25.308043902640808 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0# disk.dm-0.out‚23476.01708984375 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0bÁ. žè‚(:“ š®ŠÓ™Þ¶³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.086' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.36200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç0‚Ã6é‚(:«  ©®ŠÓ³á³! disk.dm-0.out‚23476.935546875 disk.drbd2041.in‚0 disk.drbd1285.in‚0 network.sent‚ 6.795559e+06 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚64.1875 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 180.58203125 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0( proc.memory.availableMB‚ 1944.4609375 disk.drbd1673.in‚0 disk.drbd1356.in‚0$ memory_percent‚92.08796819051108) disk./.usagePercent‚70.51330978948299 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0) proc.memory.percent‚11.73100471496582 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 cpu‚25.307989777375052 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 32.3671875 disk.nvme0n1.in‚64.1875 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 network.recv‚398630$ disk./.usageGB‚1260.849594116211 disk.drbd1289.in‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2883.01171875 proc.cpu.threads‚17 disk.drbd1965.in‚0$ disk.nvme0n1.out‚23476.935546875 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 180.58203125 disk.sda2.in‚0.03125Òϙҟê‚(:” ©®ŠÓ¥Ù³ gpu.0.gpu‚56.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚111.298' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.099333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06šàžë‚(:“ ¸®ŠÓŸò·³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.069' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.35633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿xh˜5ì‚(:ª  ¸®ŠÓªëس disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 cpu‚25.30782876947219 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚64.1875 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 network.recv‚398630% disk./.usageGB‚1260.8495750427246 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.in‚0.03125$ memory_percent‚92.08594957987467) proc.memory.percent‚11.73100471496582 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.sda.out‚ 180.890625# proc.memory.rssMB‚ 2883.01171875( disk./.usagePercent‚70.5133087227938 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚64.1875 disk.drbd1938.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1944.95703125 network.sent‚ 6.795559e+06! disk.dm-0.out‚23477.185546875$ disk.nvme0n1.out‚23477.185546875 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 180.890625 disk.rbd1.out‚ 32.3994140625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0ž^<í‚(:±  Ç®ŠÓ©îͳ disk.drbd1550.out‚0 disk.drbd1584.in‚0 network.sent‚ 6.795667e+06 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 cpu‚25.3077765441725 disk.drbd1938.in‚0 disk.nvme0n1.in‚64.1875% disk./.usageGB‚1260.8495750427246 disk.drbd1673.out‚0 disk.drbd1285.in‚0 network.recv‚398738 disk.drbd1285.out‚0& disk.nvme0n1.out‚23477.52880859375 disk.rbd1.out‚ 32.5009765625 disk.drbd1550.in‚0) proc.memory.percent‚11.73100471496582 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚64.1875 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17$ memory_percent‚92.08594957987467 disk.sda2.out‚ 181.23046875 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚23477.52880859375) proc.memory.availableMB‚ 1944.95703125 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda.out‚ 181.23046875( disk./.usagePercent‚70.5133087227938 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0ÄUÌŸî‚(:” Ç®ŠÓÊý´ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚303.493' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.16433333333332 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜î2ï- _wandbruntime‚6925  _runtime‚6925®ú‹ÇðÁ & lr-AdamW/pg2‚9.912340639977364e-05 trainer/global_step‚3799 % _timestamp‚1.7848461616471744e+09 _runtime‚6925.004398292 _step‚313 & lr-AdamW/pg1‚9.912340639977364e-05J6üÌñÆ & lr-AdamW/pg2‚9.912340639977364e-05 trainer/global_step‚3799 % _timestamp‚1.7848461616471744e+09 _runtime‚6925.004398292 _step‚313 & lr-AdamW/pg1‚9.912340639977364e-05¹~)k2ò- _wandbruntime‚6925  _runtime‚6925ìêPµ óýO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1097196420608000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32961624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4845778105856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1092800523036080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10126304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1093232185444352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 L?DeviceStatsMonitor.on_tr¯dðain_batch_start/active.small_pool.freed‚35264036 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1093232159270912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21685520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32961630 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1097195048998400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10126299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35264036 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3964235163648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11559216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11558861 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1096756776588649 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1093232159270912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3964235163648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32961630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚6925.008936876 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1097195048998400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848461616482236e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1096758146627903 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304302574781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962889727488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304302157520384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1093232185444352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68235730 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1092800548464048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21685160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299456747505664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4845827275776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 trainer/global_step‚3799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32961624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35274100 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚299456379414528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1097196420608000 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1097196420608000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32961624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4845778105856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1092800523036080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10126304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1093232185444352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35264036 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1093232159270912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21685520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32961630 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1097195048998400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10126299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35264036 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3964235163648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11559216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11558861 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1096756776588649 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1093232159270912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3964235163648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32961630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚6925.008936876 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1097195048998400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848461616482236e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1096758146627903 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304302574781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962889727488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304302157520384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1093232185444352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68235730 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1092800548464048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21685160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299456747505664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286ºõô¡$2õ- _wandbruntime‚6926  _runtime‚6926Ëâ¿ 'ö‡N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304382408606720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚6926.478912258 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35283196 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10128939 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1097036599164065 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1097476352413696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1093511068561408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299534965780480 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1093511068561408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68253193 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4847017183744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21691113 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1097474980796928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11562169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1097037969203431 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _step‚315 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21690749 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚359 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1093079324809856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3965257678848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68243109 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 % _timestamp‚1.7848461631213138e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32969991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11561810 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35273118 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35283196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3958618965607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10128944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35273118 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963912235520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3965257678848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68253193 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1093079350237824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864‘ª¥§¦÷ŒN F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32969991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11561810 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35273118 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35283196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3958618965607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10128944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35273118 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963912235520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3965257678848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68253193 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1093079350237824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚–X±l304382408606720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚6926.478912258 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35283196 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10128939 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1097036599164065 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1097476352413696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1093511068561408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299534965780480 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1093511068561408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68253193 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4847017183744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21691113 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1097474980796928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11562169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1097037969203431 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _step‚315 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21690749 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚359 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1093079324809856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3965257678848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68243109 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 % _timestamp‚1.7848461631213138e+09 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33379974 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10258304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303386065803264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848462486281996e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69110032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1111191703079424 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1111191703079424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1107177623795712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33379980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35719988 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35730052 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4908092502528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11705940 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1110747969160257 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33379980 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4908032846848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21964239 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1107177649969152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4007357435377 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1107177649969152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1111193074689024 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69099962 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1106740611724880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33379974 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10258299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚381 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286©`ŽÔÉ Œ‚P ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4015424719872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33379974 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10258304 eQDeviceStatsÅ¢a-?Monitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303386065803264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848462486281996e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69110032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1111191703079424 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1111191703079424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1107177623795712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33379980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35719988 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35730052 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4908092502528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11705940 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1110747969160257 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33379980 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4908032846848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21964239 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1107177649969152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4007357435377 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1107177649969152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1111193074689024 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69099962 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1106740611724880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33379974 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10258299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚381 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4015424719872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35730052 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1110749339199511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824¾ˆmŒL2- _wandbruntime‚7013  _runtime‚7013pƒq 'Ž‡N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303464284078080 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33388341 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1111473006494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69127495 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10260944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1111027791735673 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33388341 _runtime‚7013.657479496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4016447235072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚367 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚319 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1107456559259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10260939 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11709223 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308373551715328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35739148 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21969795 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1107019413498656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4008378237017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69127495 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10260944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1111027791735673 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33388341 _runtime‚7013.657479496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4016447235072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚367 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚319 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1107456559259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10260939 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11709223 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308373551715328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35739148 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21969795 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1107019413498656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4008378237017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33798324 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69974264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4066614276096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69984334 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36186004 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36175940 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22244296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33798324 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22243909 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10390304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11853610 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚382 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1124740531771119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36186004 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1121123088320512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚387 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33798330 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1125188357160448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1121123114493952 _step‚322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4970434161152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10390299 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36186004 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1121123088320512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚387 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33798330 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1125188357160448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1121123114493952 _step‚322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4970434161152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10390299 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33798324 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69974264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4066614276096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69984334 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36186004 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36175940 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22244296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33798324 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22243909 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10390304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11853610 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚382 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1124740531771119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22249835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1121401997611008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4067636791296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36185022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784846337179922e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4971724792832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33806691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚385 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚390 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4971665144320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4060826731223 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36185022 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312365695259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70001797 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1125469660575744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1125018984307281 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1120959502187456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1125020354346647 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1121402023784448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22249445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4059482119825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1120959527615424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69991713 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36195100 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10392944 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69991713 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33806697 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33806691 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307393602375680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36195100 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1125468288958976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4066291347968 J'§‹N UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4066291347968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1125469660575744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4067636791296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22249835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1121401997611008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4067636791296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36185022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784846337179922e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4971724792832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33806691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚385 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚390 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4971665144320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4060826731223 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36185022 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312365695259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70001797 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1125469660575744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1125018984307281 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1120959502187456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1125020354346647 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1121402023784448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22249445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4059482119825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1120959527615424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69991713 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36195100 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10392944 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69991713 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33806697 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33806691 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307393602375680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36195100 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1125468288958976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4066291347968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11856506 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11856891 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181Ã÷ê¢ß2¨- _wandbruntime‚7100  _runtime‚7100=»â’©þ -loss/FRt2/detection_loss‚1.51820063841926 _step‚324 1loss/FBt2/assignment_loss‚0.06281707956846759 0loss/SRqqt1/detection_loss‚0.905115287261454 1loss/SRqqt2/detection_loss‚0.6843895380154031 epoch‚13 % _timestamp‚1.7848463372145755e+09 _runtime‚7100.571820089 /loss/FBt2/detection_loss‚0.5113614038886302 2loss/SRqqt1/assignment_loss‚0.6917893343658312 2loss/FBt1/assignment_loss‚0.003679397139892209 /loss/FBt1/detection_loss‚0.9862591097580975 trainer/global_step‚3899 %loss/total_loss‚9.274240262498875 /loss/FRt1/assignment_loss‚1.935237437427517 /loss/FRt2/assignment_loss‚0.796953490238316 2loss/SRqqt2/assignment_loss‚0.6156407195851581 /loss/FRt1/detection_loss‚0.5627968268308472?A‰ ªƒ % _timestamp‚1.7848463372145755e+09 _runtime‚7100.571820089 /loss/FBt2/detection_loss‚0.5113614038886302 2loss/SRqqt1/assignment_loss‚0.6917893343658312 2loss/FBt1/assignment_loss‚0.003679397139892209 /loss/FBt1/detection_loss‚0.9862591097580975 trainer/global_step‚3899 %loss/total_loss‚9.274240262498875 /loss/FRt1/assignment_loss‚1.935237437427517 /loss/FRt2/assignment_loss‚0.796953490238316 2loss/SRqqt2/assignment_loss‚0.6156407195851581 /loss/FRt1/detection_loss‚0.5627968268308472 -loss/FRt2/detection_loss‚1.51820063841926 _step‚324 1loss/FBt2/assignment_loss‚0.06281707956846759 0loss/SRqqt1/detection_loss‚0.905115287261454 1loss/SRqqt2/detection_loss‚0.6843895380154031 epoch‚13ĸFô"ž«‚(:“ Š°ŠÓ·£¨³ gpu.0.gpu‚58.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.476' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.82533333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ >ýB¬‚(:·  Š°ŠÓé»Ð³ disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.dm-0.in‚ 66.06640625% disk./.usageGB‚1260.8502578735352% disk.nvme0n1.out‚23482.2080078125 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.08442370096842) proc.memory.availableMB‚ 1945.33203125 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 66.06640625 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 33.4560546875 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0" disk.dm-0.out‚23482.2080078125 disk.drbd2111.out‚0 cpu‚25.307004184414023 disk.drbd2118.in‚0 disk.sda2.out‚ 186.4921875 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2883.01171875) proc.memory.percent‚11.73100471496582 network.sent‚ 6.952528e+06 network.recv‚407278( disk./.usagePercent‚70.5133469102673 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda.out‚ 186.4921875 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0²1) ž­‚(:“ ™°ŠÓœÔ̳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.585' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.52833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ¡ÐA®‚(:¶  ™°ŠÓÌÑí³ disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.sda.out‚ 186.9765625 disk.drbd1938.out‚0 network.recv‚409835 disk.drbd1550.in‚0 disk.drbd2111.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 66.06640625 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 186.9765625 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0% disk.nvme0n1.out‚23482.9111328125$ memory_percent‚92.08855628967285% disk./.usageGB‚1260.8501625061035 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0) proc.memory.percent‚11.73100471496582 network.sent‚ 7.01291e+06) disk./.usagePercent‚70.51334157682129 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1944.31640625 cpu‚25.30688306302956" disk.dm-0.out‚23482.9111328125 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 33.6357421875 disk.dm-0.in‚ 66.06640625YD[ˆœ¯‚(:‘ ¨°ŠÓá°Ö³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.065' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.68833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò;¼G;°‚(:°  ¨°ŠÓ™¤ö³ disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1550.out‚0) proc.memory.percent‚11.73100471496582" disk.dm-0.out‚23483.2431640625 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 cpu‚25.30826213719171) disk./.usagePercent‚70.51334605691594$ disk./.usageGB‚1260.850242614746 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 33.6982421875$ memory_percent‚92.08760261535645 network.sent‚ 7.013887e+06 network.recv‚410688 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 66.06640625 disk.sda1.out‚0 disk.dm-0.in‚ 66.06640625 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0% disk.nvme0n1.out‚23483.2431640625 disk.sda.out‚187.4375 disk.drbd1931.out‚0 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2883.01171875 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 1944.55078125 disk.drbd2041.in‚0 disk.sda2.out‚187.4375 proc.cpu.threads‚17§(vÐ2±- _wandbruntime‚7154  _runtime‚7154ÂU»-§²Öv ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6503323912620544 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7452818751335144 )loss/val_total_loss‚9.605020923349088 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4564224183559418 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11502528190612793 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18463429808616638 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8609841465950012 particle/accuracy_2_of_3‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8984583616256714 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7425157427787781 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3265567421913147 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2948271632194519 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6623808741569519 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23098424077033997 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1849707067012787 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9061105847358704 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6671934723854065 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3358171582221985 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9983766674995422 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19536474347114563 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.645416796207428 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23513385653495789 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8064332008361816 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9236111044883728 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08140386641025543 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.904552161693573 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39500942826271057 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6307616829872131 epoch‚13 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1527777761220932 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1820938140153885 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6593652963638306 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7556396722793579 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5846153497695923 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9236111044883728 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996642529964447 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8197073936462402 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8137136697769165 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23513385653495789 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.663884162902832 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22706541419029236 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13241465389728546 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9984472393989563 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8140744566917419 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44774574041366577 *jet/accuracy_1_of_4‚0.9803374409675598 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_2‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04068777337670326 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21966834366321564 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4249802231788635 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958745241165161 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6419214606285095 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7567495703697205 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8052540421485901 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19679081439971924 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6535776853561401 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 *jet/accuracy_1_of_2‚0.7845056653022766 *jet/accuracy_2_of_2‚0.1540679782629013 +jet/accuracy_3_of_4‚0.11347367614507675 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6626871228218079 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 $ _timestamp‚1.784846391477898e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9989413022994995 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.81992107629776 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04268922284245491 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7084067463874817 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48680415749549866 *jet/accuracy_2_of_4‚0.8299686312675476 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7513830065727234 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7230912446975708 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8305901885032654 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 *particle/sensitivity‚0.628746509552002 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435380578041077 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 _step‚325 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7478553652763367 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3955588638782501 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7534176707267761 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2338511198759079 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7047394514083862 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.038595445454120636 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03641989082098007 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/0FR©EÛµt0SRqqt*FBt/FBt_purity‚0.9985564351081848 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7178577780723572 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5819407105445862 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9928570985794067 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6671934723854065 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9982576370239258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21639946103096008 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5438748002052307 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7135710716247559 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6853042244911194 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977678656578064 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1012914851307869 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20899847149848938 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04883302375674248 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7461538314819336 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9878140091896057 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7055627703666687 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1527777761220932 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 *jet/accuracy_1_of_3‚0.9258111715316772 6validation_average_jet_accuracy‚0.3758911192417145 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6891722679138184 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1527777761220932 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6591734290122986 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992559552192688 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24704387784004211 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02111036516726017 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4268653988838196 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7455776333808899 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20962783694267273 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24863018095493317 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1527777761220932 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 trainer/global_step‚3919 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5595079660415649 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041556090116500854 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1912735551595688 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7375776171684265 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 /particle/accuracy_4_of_4‚0.9680764079093933 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990968644618988 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893692135810852 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6005417704582214 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20975828170776367 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11828763037919998 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6436429619789124 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6266483068466187 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8609841465950012 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8692386746406555 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04282739758491516 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25525349378585815 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7021883726119995 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5923001170158386 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9867501854896545 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3286447823047638 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23453548550605774 'particle/f_score‚0.6375212073326111 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24916820228099823 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04003044590353966 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03641989082098007 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8466041684150696 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988378882408142 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18118150532245636 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7537914514541626 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06446121633052826 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4185410141944885 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2932884097099304 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5865859389305115 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6793609857559204 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04080318287014961 _runtime‚7154.836001574 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0B<Ôa;³Ûv particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8466041684150696 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988378882408142 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18118150532245636 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7537914514541626 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06446121633052826 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4185410141944885 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2932884097099304 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5865859389305115 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6793609857559204 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04080318287014961 _runtime‚7154.836001574 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6503323912620544 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7452818751335144 )loss/val_total_loss‚9.605020923349088 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4564224183559418 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11502528190612793 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18463429808616638 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8609841465950012 particle/accuracy_2_of_3‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8984583616256714 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7425157427787781 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3265567421913147 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2948271632194519 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6623808741569519 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23098424077033997 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1849707067012787 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9061105847358704 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6671934723854065 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3358171582221985 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9983766674995422 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19536474347114563 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.645416796207428 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23513385653495789 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8064332008361816 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9236111044883728 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08140386641025543 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.904552161693573 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39500942826271057 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6307616829872131 epoch‚13 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1527777761220932 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1820938140153885 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6593652963638306 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7556396722793579 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5846153497695923 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9236111044883728 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996642529964447 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8197073936462402 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8137136697769165 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23513385653495789 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.663884162902832 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22706541419029236 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13241465389728546 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9984472393989563 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8140744566917419 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44774574041366577 *jet/accuracy_1_of_4‚0.9803374409675598 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_2‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04068777337670326 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21966834366321564 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4249802231788635 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958745241165161 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6419214606285095 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7567495703697205 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8052540421485901 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19679081439971924 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6535776853561401 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 *jet/accuracy_1_of_2‚0.7845056653022766 *jet/accuracy_2_of_2‚0.1540679782629013 +jet/accuracy_3_of_4‚0.11347367614507675 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6626871228218079 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 $ _timestamp‚1.784846391477898e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9989413022994995 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.81992107629776 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04268922284245491 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7084067463874817 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48680415749549866 *jet/accuracy_2_of_4‚0.8299686312675476 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7513830065727234 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7230912446975708 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8305901885032654 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 *particle/sensitivity‚0.628746509552002 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435380578041077 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 _step‚325 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7478553652763367 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3955588638782501 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7534176707267761 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2338511198759079 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7047394514083862 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.038595445454120636 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03641989082098007 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9985564351081848 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7178577780723572 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5819407105445862 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9928570985794067 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6671934723854065 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9982576370239258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21639946103096008 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5438748002052307 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7135710716247559 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6853042244911194 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977678656578064 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1012914851307869 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20899847149848938 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04883302375674248 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7461538314819336 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9878140091896057 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7055627703666687 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1527777761220932 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 *jet/accuracy_1_of_3‚0.9258111715316772 6validation_average_jet_accuracy‚0.3758911192417145 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6891722679138184 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1527777761220932 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6591734290122986 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992559552192688 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24704387784004211 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02111036516726017 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4268653988838196 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7455776333808899 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20962783694267273 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24863018095493317 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1527777761220932 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 trainer/global_step‚3919 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5595079660415649 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041556090116500854 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1912735551595688 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7375776171684265 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 /particle/accuracy_4_of_4‚0.9680764079093933 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990968644618988 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893692135810852 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6005417704582214 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20975828170776367 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11828763037919998 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6436429619789124 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6266483068466187 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8609841465950012 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8692386746406555 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04282739758491516 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25525349378585815 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7021883726119995 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5923001170158386 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9867501854896545 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3286447823047638 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23453548550605774 'particle/f_score‚0.6375212073326111 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24916820228099823 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04003044590353966 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03641989082098007 particle/accuracy_3_of_3‚1ŽôÃG‘´‚(:† ·°ŠÓ޾ų gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚59.0 gpu.0.powerWatts‚106.662' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð†5¥8µ‚(:­  ·°ŠÓíûå³ disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 cpu‚25.31346242475713 disk.drbd1285.in‚0 disk.drbd1938.out‚0$ disk./.usageGB‚1260.850284576416 disk.rbd1.out‚ 33.7998046875 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0# memory_percent‚92.0887311299642 disk.drbd1828.out‚0 disk.dm-0.out‚ 23484.0703125( disk./.usagePercent‚70.5133484036322 disk.sda.out‚ 187.7734375 disk.sda2.in‚0.03125) proc.memory.percent‚11.73100471496582 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 187.7734375 disk.drbd1931.in‚0 disk.drbd1965.in‚0 network.recv‚411576 disk.drbd1552.in‚0 disk.drbd1584.in‚0 network.sent‚ 7.016333e+06 disk.nvme0n1.in‚ 66.06640625" disk.nvme0n1.out‚ 23484.0703125 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0( proc.memory.availableMB‚ 1944.2734375# proc.memory.rssMB‚ 2883.01171875 disk.drbd1226.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 66.06640625 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0éjÀ¿ž¶‚(:“ Æ°ŠÓ¡·³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚295.247' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.41566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¬F·‚(:»  Æ°ŠÓ¨‚Ô³ disk.drbd1965.out‚0 disk.sda2.out‚ 188.21484375 disk.drbd1938.in‚0 disk.drbd1828.out‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd2111.in‚0 disk.drbd1285.in‚0 network.sent‚ 7.072921e+06% disk./.usageGB‚1260.8540496826172 disk.sda.out‚ 188.21484375# disk.dm-0.out‚24453.94189453125 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 66.06640625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 network.recv‚413164) disk./.usagePercent‚70.51355896808111 disk.drbd1931.in‚0 disk.rbd1.out‚ 33.8623046875 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0( proc.memory.availableMB‚ 2916.1015625 proc.cpu.threads‚17 disk.sda1.out‚0$ memory_percent‚88.13435236612955 disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚24453.94189453125 disk.drbd2118.in‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.730988820393879 disk.dm-0.in‚ 66.06640625 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1510.out‚0 cpu‚25.311526650819783 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0vÒ44ž¸‚(:“ Õ°ŠÓàá´³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚295.591' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.53033333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÍKA¹‚(:¶  Õ°ŠÓ˜ŠÖ³ disk.drbd1356.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1931.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 188.9140625 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚24455.1005859375 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 network.sent‚ 7.075913e+06 disk.drbd2041.in‚0 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1226.in‚0 disk.drbd1552.out‚0( disk./.usagePercent‚70.5135568347027 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 33.8623046875 disk.drbd1285.out‚0 disk.drbd1673.out‚0$ memory_percent‚88.13435236612955 disk.sda2.out‚ 188.9140625 disk.drbd1931.out‚0 cpu‚25.311502589014708 disk.nvme0n1.in‚ 66.06640625 network.recv‚413991% disk./.usageGB‚1260.8540115356445 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125( proc.memory.availableMB‚ 2916.1015625 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0" disk.dm-0.out‚24455.1005859375 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.dm-0.in‚ 66.06640625 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.730988820393879 proc.cpu.threads‚17/!@º‚(:µ  ä°ŠÓ¿±é³ disk.sda.out‚ 189.22265625 disk.drbd2078.out‚0$ memory_percent‚88.13435236612955 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.dm-0.in‚ 66.06640625 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 network.sent‚ 7.078894e+06% disk.nvme0n1.out‚25424.9443359375 disk.drbd1289.out‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51334435021322$ disk./.usageGB‚1260.850212097168 disk.nvme0n1.in‚ 66.06640625 disk.drbd1550.in‚0 disk.drbd1356.in‚0 cpu‚25.31137393992529 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 33.89453125 disk.sda1.out‚0 disk.sda2.in‚0.03125 network.recv‚414795 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.sda2.out‚ 189.22265625 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.drbd1510.in‚0 disk.drbd1938.in‚0" disk.dm-0.out‚25424.9443359375( proc.memory.availableMB‚ 2916.1015625 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0" proc.memory.rssMB‚ 2883.0078125* proc.memory.percent‚11.730988820393879 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0G‘kÉž»‚(:“ ä°ŠÓБʳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚278.325' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.77499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã!ª2¼- _wandbruntime‚7210  _runtime‚7210fØ TǽÁ & lr-AdamW/pg1‚9.764873108849978e-05 & lr-AdamW/pg2‚9.764873108849978e-05 trainer/global_step‚3949 % _timestamp‚1.7848464474013264e+09 _runtime‚7210.758567933 _step‚326DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1141751306405376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36679204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70995380 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5040167049728 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4123422271488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10529191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311293333820416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚322 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36679204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚317 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1141294964293955 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4116518651593 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36689268 % _timestamp‚1.7848464474024086e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4122076835328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1137629229570048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22558364 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34306106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚7210.762944816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36689268 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1141752678014976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1141751306405376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1137629229570048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10529196<6%›(Á‚P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316333868961280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36679204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚317 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1141294964293955 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4116518651593 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36689268 % _timestamp‚1.7848464474024086e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4122076835328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1137629229570048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22558364 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34306106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚7210.762944816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36689268 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1141752678014976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1141751306405376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1137629229570048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10529196 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1141751306405376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36679204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70995380 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5040167049728 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4123422271488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10529191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311293333820416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚322 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22563609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10531836 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34314479 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4124444786688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4116194841947 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1137908165033984 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1137458617455392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71002759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36698364 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12032091 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1141574786869371 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36698364 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5041428971008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1137908138860544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36688286 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316413717248512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34314473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311371920186368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1142031238203904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34314473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4124444786688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1137458592027424 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1137908165033984 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1137908138860544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4123099343360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5041386099712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36688286 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71012843 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316413306286080 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71012843 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 trainer/global_step‚3949 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848464488791797e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311372288277504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1141576156908737 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71002759 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22563927 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4123099343360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12031778 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚7212.236844663 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0+£ò`³ÄŒN [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1137458592027424 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1137908165033984 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1137908138860544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4123099343360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5041386099712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36688286 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71012843 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316413306286080 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71012843 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 trainer/global_step‚3949 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848464488791797e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311372288277504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1141576156908737 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71002759 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22563927 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4123099343360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12031778 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚7212.236844663 FDeviceStatsMonitor.on_traâ`_in_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22563609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10531836 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34314479 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4124444786688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4116194841947 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1137908165033984 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1137458617455392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71002759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36698364 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12032091 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1141574786869371 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36698364 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5041428971008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1137908138860544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36688286 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316413717248512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34314473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311371920186368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1142031238203904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34314473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4124444786688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224ÈŸm³Ý2Å- _wandbruntime‚7212  _runtime‚7212ÓýTËÆ‚ 1loss/FBt2/assignment_loss‚0.04035019116219187 _runtime‚7212.272870512 /loss/FRt1/detection_loss‚0.5615432259561203 /loss/FRt2/detection_loss‚1.4232108403352348 /loss/FBt2/detection_loss‚0.8015597264554719 %loss/total_loss‚8.885993534972942 epoch‚14 % _timestamp‚1.7848464489156032e+09 _step‚329 0loss/FRt1/assignment_loss‚1.9554730673631033 0loss/SRqqt1/assignment_loss‚0.57152312759242 2loss/SRqqt2/assignment_loss‚0.5166088368795582 1loss/SRqqt1/detection_loss‚0.8600484092989568 3loss/FBt1/assignment_loss‚0.0023990130660739135 1loss/SRqqt2/detection_loss‚0.6238349242329493 /loss/FBt1/detection_loss‚0.7544401371186333 trainer/global_step‚3949 0loss/FRt2/assignment_loss‚0.7750020355122275CÇk LJ 0loss/FRt2/assignment_loss‚0.7750020355122275 1loss/FBt2/assignment_loss‚0.04035019116219187 _runtime‚7212.272870512 /loss/FRt1/detection_loss‚0.5615432259561203 /loss/FRt2/detection_loss‚1.4232108403352348 /loss/FBt2/detection_loss‚0.8015597264554719 %loss/total_loss‚8.885993534972942 epoch‚14 % _timestamp‚1.7848464489156032e+09 _step‚329 0loss/FRt1/assignment_loss‚1.9554730673631033 0loss/SRqqt1/assignment_loss‚0.57152312759242 2loss/SRqqt2/assignment_loss‚0.5166088368795582 1loss/SRqqt1/detection_loss‚0.8600484092989568 3loss/FBt1/assignment_loss‚0.0023990130660739135 1loss/SRqqt2/detection_loss‚0.6238349242329493 /loss/FBt1/detection_loss‚0.7544401371186333 trainer/global_step‚3949Éã—D4È‚(:©  ó°ŠÓ”§ç³ network.sent‚ 7.081437e+06 disk.drbd1828.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.730988820393879 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.sda.out‚189.6875 disk.drbd1673.out‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 2915.9609375 disk.drbd1552.out‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1828.in‚0 disk.rbd1.out‚ 33.99609375 disk.rbd1.in‚0) disk./.usagePercent‚70.51334435021322 disk.dm-0.in‚ 66.06640625 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 cpu‚25.31131610798625 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚189.6875 disk.drbd1938.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0$ memory_percent‚88.13492457071939 network.recv‚415770$ disk./.usageGB‚1260.850212097168 disk.nvme0n1.in‚ 66.06640625 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.dm-0.out‚25425.50390625# disk.nvme0n1.out‚25425.50390625 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0_Yæ‚ŸÉ‚(:” ó°ŠÓ«ô³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.084' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.36133333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ç$žÊ‚(:“ ‚±ŠÓÞéƳ gpu.0.gpu‚37.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.64' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.21333333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ªP?Ë‚(:´  ‚±ŠÓáæ³ disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 66.06640625 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 190.05859375 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 cpu‚25.311327168252305% disk./.usageGB‚1260.8502311706543 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0) disk./.usagePercent‚70.51334541690241 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 34.05859375 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 190.05859375 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.dm-0.out‚25425.73828125# disk.nvme0n1.out‚25425.73828125 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.sda.in‚0.03125 disk.rbd1.in‚0 network.recv‚417659 disk.drbd1817.in‚0 disk.drbd2111.in‚0$ memory_percent‚88.13630739847818" proc.memory.rssMB‚ 2883.0078125 disk.dm-0.in‚ 66.06640625) proc.memory.availableMB‚ 2915.62109375 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.730988820393879 network.sent‚ 7.141445e+06¢kúµ“Ì‚(:ˆ ‘±ŠÓ¡ãϳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.699' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.233 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hTÓ@Í‚(:µ  ‘±ŠÓÚ÷è³ disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1584.out‚0 network.recv‚418982 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 66.06640625 disk.drbd1550.in‚0 cpu‚25.31123496524234 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 66.06640625 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 2916.11328125) disk./.usagePercent‚70.51330210932073 disk.drbd1510.out‚0% disk./.usageGB‚1260.8494567871094 disk.drbd1931.out‚0 network.sent‚ 7.145298e+06 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 34.05859375 disk.drbd1285.in‚0 disk.sda.out‚ 190.5234375$ memory_percent‚88.13430468241373" proc.memory.rssMB‚ 2883.0078125 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 190.5234375 disk.drbd1289.out‚0" disk.dm-0.out‚25426.0244140625% disk.nvme0n1.out‚25426.0244140625 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.730988820393879 disk.drbd1965.out‚0 disk.drbd1828.in‚09‰›dŸÎ‚(:”  ±ŠÓÏ©¬³ gpu.0.gpu‚68.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.703' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.23433333333331 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™üãhAÏ‚(:¶   ±ŠÓ­ë˳ disk.drbd1938.out‚0" disk.dm-0.out‚25426.1845703125 disk.drbd1226.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 34.0908203125 disk.drbd2078.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.730988820393879& proc.memory.availableMB‚ 2916.09375 cpu‚25.311282051146645$ disk./.usageGB‚1260.849479675293 disk.drbd1552.in‚0 disk.drbd1828.out‚0 network.recv‚425820 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0$ memory_percent‚88.13438415527344 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0 network.sent‚ 7.151494e+06 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 66.06640625 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚25426.1845703125 disk.drbd1584.in‚0) disk./.usagePercent‚70.51330338934777 disk.nvme0n1.in‚ 66.06640625 disk.sda.out‚ 190.73828125" proc.memory.rssMB‚ 2883.0078125 disk.sda2.out‚ 190.73828125 disk.drbd1285.out‚0 disk.drbd1510.out‚0ef“Ђ(:ˆ ¯±ŠÓœ”á³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.496' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.832 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OœfnBÑ‚(:·  ¯±ŠÓÀÔ´ disk.drbd2118.out‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 network.recv‚425886 disk.drbd2078.out‚0* proc.memory.percent‚11.730988820393879 disk.drbd1289.out‚0 disk.drbd1965.out‚0$ memory_percent‚88.13438415527344 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1550.in‚0 cpu‚25.311153252879546 disk.drbd1552.in‚0 disk.sda.out‚ 191.08984375 disk.rbd1.in‚0 disk.rbd1.out‚ 34.2783203125 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1356.in‚0 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 191.08984375 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 network.sent‚ 7.15156e+06 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚25426.58056640625 disk.drbd1510.out‚0& proc.memory.availableMB‚ 2916.09375 disk.drbd1817.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51330338934777$ disk./.usageGB‚1260.849479675293" proc.memory.rssMB‚ 2883.0078125 disk.sda1.in‚0# disk.dm-0.out‚25426.58056640625 disk.drbd2111.in‚0Ë%!þÒ‚(:’ ¾±ŠÓàøÚ³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.87' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.28999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žâú'?Ó‚(:´  ¾±ŠÓ±•÷³ disk.drbd1931.in‚0 disk.drbd1289.out‚0$ disk./.usageGB‚1260.849479675293 disk.drbd1552.out‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1510.in‚0 disk.sda.in‚0.03125 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.rbd1.out‚ 34.3408203125) proc.memory.availableMB‚ 1944.13671875* proc.memory.percent‚11.730988820393879 disk.sda2.out‚ 191.328125 network.recv‚427191 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 70.68359375# disk.dm-0.out‚25426.82666015625 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda.out‚ 191.328125 disk.sda2.in‚0.03125) disk./.usagePercent‚70.51330338934777 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 cpu‚25.3111309811623 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0$ memory_percent‚92.08928743998209 network.sent‚ 7.15222e+06 disk.drbd1552.in‚0 disk.drbd1584.out‚0& disk.nvme0n1.out‚25426.82666015625 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚09Ÿ|2Ô- _wandbruntime‚7297  _runtime‚7297q•’•ÇÕÁ & lr-AdamW/pg1‚9.714683996676392e-05 & lr-AdamW/pg2‚9.714683996676392e-05 trainer/global_step‚3999 % _timestamp‚1.7848465344233863e+09 _runtime‚7297.780619451 _step‚330÷ÿÐÌÖÆ % _timestamp‚1.7848465344233863e+09 _runtime‚7297.780619451 _step‚330 & lr-AdamW/pg1‚9.714683996676392e-05 & lr-AdamW/pg2‚9.714683996676392e-05 trainer/global_step‚3999ÊkN62×- _wandbruntime‚7297  _runtime‚7297œ@¥((ØýO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1151119904370416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320325875981312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1151574720268288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4173266391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12176417 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚3999 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34724456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚331 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34724462 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155287526904817 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10661191 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315222652118016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1151119878942448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5102855772160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71869682 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320325450331648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1151574720268288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37135156 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1155747960486400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4166277923115 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1155749332096000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22837613 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7297.781974038 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070Á©I)²Ù‚P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37145220 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34724462 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155287526904817 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10661191 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315222652118016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1151119878942448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.alloµsø&Vcated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5102855772160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71869682 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320325450331648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1151574720268288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37135156 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1155747960486400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4166277923115 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1155749332096000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22837613 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7297.781974038 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1151119904370416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320325875981312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1151574720268288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4173266391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12176417 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚3999 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34724456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚331 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 Ià 'Û‡N G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71887145 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34732829 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71877061 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34732823 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37154316 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1156029263901696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10663836 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71877061 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4174288899584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1151398680716224 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4168643336153 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1156029263901696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚341 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315301238483968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1156027892284928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22843177 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5104056194560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 % _timestamp‚1.7848465359025464e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71887145 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1155565979440979 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1151398706144192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 _step‚332 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4175634342912 trainer/global_step‚3999 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1151853603385344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320405714029568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37144238 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22842836 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1151853629558784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10663831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12179005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315301606575104 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1155567349480345 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37144238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1151853603385344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22842836 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1151853629558784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10663831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12179005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315301606575104 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1155567349480345 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37144238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1151853603385344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34732829 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71877061 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34732823 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37154316 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1156029263901696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10663836 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71877061 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4174288899584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1151398680716224 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4168643336153 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1156029263901696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚341 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315301238483968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1156027892284928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22843177 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5104056194560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 % _timestamp‚1.7848465359025464e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71887145 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1155565979440979 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1151398706144192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 _step‚332 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4175634342912 trainer/global_step‚3999 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1151853603385344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Ìr`¬2Ý- _wandbruntime‚7299  _runtime‚7299\6öHÞý _runtime‚7299.295202235 %loss/total_loss‚9.470161397049388 2loss/SRqqt1/assignment_loss‚0.6024589886085244 2loss/SRqqt2/assignment_loss‚0.5054145299646069 /loss/FRt2/detection_loss‚1.4762394690197431 1loss/SRqqt2/detection_loss‚0.6509872854763347 .loss/FBt1/detection_loss‚1.076530096151175 trainer/global_step‚3999 % _timestamp‚1.7848465359379928e+09 0loss/FRt1/assignment_loss‚1.8633749772921913 0loss/FBt2/assignment_loss‚0.0546312020332034 _step‚333 .loss/FBt2/detection_loss‚0.514828544457666 0loss/FRt2/assignment_loss‚0.8751312932547402 0loss/FBt1/assignment_loss‚0.2508104633027827 /loss/FRt1/detection_loss‚0.5462186599167266 0loss/SRqqt1/detection_loss‚1.053535887571694 epoch‚145…ß‚ .loss/FBt2/detection_loss‚0.514828544457666 0loss/FRt2/assignment_loss‚0.8751312932547402 0loss/FBt1/assignment_loss‚0.2508104633027827 /loss/FRt1/detection_loss‚0.5462186599167266 0loss/SRqqt1/detection_loss‚1.053535887571694 epoch‚14 _runtime‚7299.295202235 %loss/total_loss‚9.470161397049388 2loss/SRqqt1/assignment_loss‚0.6024589886085244 2loss/SRqqt2/assignment_loss‚0.5054145299646069 /loss/FRt2/detection_loss‚1.4762394690197431 1loss/SRqqt2/detection_loss‚0.6509872854763347 .loss/FBt1/detection_loss‚1.076530096151175 trainer/global_step‚3999 % _timestamp‚1.7848465359379928e+09 0loss/FRt1/assignment_loss‚1.8633749772921913 0loss/FBt2/assignment_loss‚0.0546312020332034 _step‚333ÍNüUðœà‚(:‘ ͱŠÓ¬ŽÇ³ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚278.558' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.85266666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%½æFá‚(:»  ͱŠÓʼné³ disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 network.sent‚ 7.157811e+06 disk.rbd1.in‚0( proc.memory.availableMB‚ 1944.1015625 disk.drbd2041.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 network.recv‚428229 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 70.68359375& disk.nvme0n1.out‚25427.08154296875 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1965.in‚0$ memory_percent‚92.08943049112955 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1931.out‚0 cpu‚25.311074963890878 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 191.77734375 disk.rbd1.out‚ 34.3408203125 disk.drbd1550.out‚0* proc.memory.percent‚11.730988820393879 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚25427.08154296875) disk./.usagePercent‚70.51341091161956 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0" proc.memory.rssMB‚ 2883.0078125% disk./.usageGB‚1260.8514022827148 disk.drbd2111.out‚0 disk.sda.out‚ 191.77734375 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0UI=ážâ‚(:“ ܱŠÓÀçµ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.907' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.30233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0" ­Oã‚(:¸  ܱŠÓÒ¡Ú³# disk.dm-0.out‚25427.30419921875 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.rbd1.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚25427.30419921875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1550.in‚0( disk./.usagePercent‚70.5134111249574 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 network.sent‚ 7.160903e+06$ disk./.usageGB‚1260.851ŸØËô406097412 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 cpu‚25.310951206738448 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.730988820393879 disk.rbd1.out‚ 34.373046875 disk.drbd1510.in‚0 disk.sda.out‚ 192.05078125 disk.drbd1226.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.dm-0.in‚ 70.68359375 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 192.05078125 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 70.68359375( proc.memory.availableMB‚ 1944.1015625" proc.memory.rssMB‚ 2883.0078125 network.recv‚429227 disk.drbd1931.out‚0 disk.drbd1965.in‚0$ memory_percent‚92.08943049112955 disk.drbd1356.out‚0 disk.drbd1550.out‚0¹÷ä‚(:’ 뱊ÓÁ»½³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.64' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.54666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̵…é8å‚(:­  뱊Ó²ÎÞ³ disk.drbd1285.in‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51336717736218% disk./.usageGB‚1260.8506202697754 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 network.recv‚430901 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1510.out‚0$ memory_percent‚92.08985964457193 disk.drbd1550.out‚0 disk.sda.out‚ 192.4609375 disk.rbd1.in‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2883.0078125* proc.memory.percent‚11.730988820393879 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 192.4609375 disk.drbd1356.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 34.568359375 disk.drbd1584.in‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1828.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 network.sent‚ 7.221007e+06 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.dm-0.out‚ 25427.71875 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.out‚ 25427.71875) proc.memory.availableMB‚ 1943.99609375 cpu‚25.310860814705016Õ> ¥?æ‚(:´  ú±ŠÓ¼ôÿ³$ disk./.usageGB‚1260.850658416748 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.730988820393879 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.dm-0.out‚25427.84765625 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0$ memory_percent‚92.08985964457193 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0" proc.memory.rssMB‚ 2883.0078125 network.sent‚ 7.221073e+06 disk.rbd1.out‚ 34.630859375 disk.drbd1552.out‚0 disk.sda.out‚ 193.19140625) disk./.usagePercent‚70.51336931074059 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd2041.in‚0 network.recv‚430967 disk.sda2.out‚ 193.19140625 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 cpu‚25.310803023160247 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1943.99609375 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0# disk.nvme0n1.out‚25427.84765625 disk.drbd1817.in‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1965.in‚0G§øs“ç‚(:ˆ ú±ŠÓ‡æܳ gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚109.254' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.418 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÈ?žè‚(:“ ‰²ŠÓà®Ô³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚274.645' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.54833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜a¶3?é‚(:´  ‰²ŠÓôïñ³ disk.drbd1828.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 193.7265625 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚25428.6044921875 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 70.68359375 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0 network.sent‚ 7.22979e+06 disk.drbd1673.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0" disk.dm-0.out‚25428.6044921875 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.sda.out‚ 193.7265625 disk.drbd2078.out‚0 disk.rbd1.out‚ 34.630859375 disk.drbd1289.in‚0 disk.sda2.in‚0.03125 disk.drbd1584.out‚0 network.recv‚433664% disk./.usageGB‚1260.8515586853027 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0" memory_percent‚92.087189356486 cpu‚25.310680539099398 disk.drbd1285.out‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1965.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1944.65234375* proc.memory.percent‚11.730988820393879 disk.dm-0.in‚ 70.68359375 disk.drbd1356.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51341965847104ZÄ7Bê‚(:·  ˜²ŠÓƳų" disk.dm-0.out‚25428.8662109375 disk.drbd2041.out‚0% disk.nvme0n1.out‚25428.8662109375 disk.sda.in‚0.03125 cpu‚25.310593988249728 proc.cpu.threads‚17) disk./.usagePercent‚70.51343117871444 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd1938.out‚0( proc.memory.availableMB‚ 1944.8984375* proc.memory.percent‚11.730988820393879 network.sent‚ 7.230767e+06 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1552.in‚0 network.recv‚434665 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0$ memory_percent‚92.08618799845377 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 34.873046875 disk.drbd1285.in‚0 disk.dm-0.in‚ 70.68359375$ disk./.usageGB‚1260.851764678955 disk.drbd1817.in‚0 disk.sda2.out‚ 194.02734375 disk.drbd1289.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 194.02734375RˆÞë‚(:“ ˜²ŠÓߨ³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.362' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.78733333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ–B 2ì- _wandbruntime‚7384  _runtime‚7384„K%†ÇíÁ _runtime‚7384.834703821 _step‚334 & lr-AdamW/pg1‚9.663989158573358e-05 & lr-AdamW/pg2‚9.663989158573358e-05 trainer/global_step‚4049 % _timestamp‚1.7848466214774566e+09÷¹ÌîÆ & lr-AdamW/pg1‚9.663989158573358e-05 & lr-AdamW/pg2‚9.663989158573358e-05 trainer/global_step‚4049 % _timestamp‚1.7848466214774566e+09 _runtime‚7384.834703821 _step‚334ÎëáÛC2ï- _wandbruntime‚7384  _runtime‚7384 ¢(ðýO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35142806 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4225801383936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚324317601721344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37591108 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4224455947776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37591108 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1169277349437171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚357 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1169745986177024 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1165520158619648 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35142806 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4218726417209 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72743984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10793191 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1169744614567424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4217381805923 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12321649 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5165209850368 _runtime‚7384.839945764 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5165263214592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1165059967631248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324317180265984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1165520158619648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1169745986177024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10793196 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23114840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1169745986177024 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1165520158619648 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35142806 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4218726417209 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72743984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10793191 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1169744614567424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4217381805923 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12321649 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5165209850368 _runtime‚7384.839945764 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5165263214592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1165059967631248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324317180265984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1165520158619648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1169745986177024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10793196 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23114840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848466229569619e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35151179 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4219747218961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10795831 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1170025917982720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5166517100544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10795836 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1165338794832992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitoI/¿(r.on_train_batch_end/inactive_split.all.current‚350 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37600190 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4226823899136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1165799067910144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4225478455808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37600190 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 trainer/global_step‚4049 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37610268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1169558542051953 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319230924872704 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35151173 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1165799094083584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324397441973248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72761447 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1165799067910144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35151173 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37610268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12324889 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72751363 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23120375 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1165799094083584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324397018427904 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4225478455808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5166461646336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35151179 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12324544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72761447 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37600190 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4226823899136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1165799067910144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4225478455808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37600190 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 trainer/global_step‚4049 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37610268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1169558542051953 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319230924872704 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35151173 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1165799094083584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324397441973248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72761447 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1165799067910144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35151173 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37610268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12324889 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72751363 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23120375 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1165799094083584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324397018427904 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4225478455808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5166461646336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35151179 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12324544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72761447 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848466229569619e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35151179 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4219747218961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10795831 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1170025917982720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5166517100544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10795836 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÐO‚›2õ- _wandbruntime‚7386  _runtime‚7386[{"˜öÿ _step‚337 0loss/FRt1/assignment_loss‚1.9112284068623417 1loss/FBt2/assignment_loss‚0.04800093995527983 /loss/FRt1/detection_loss‚0.5944387452569273 /loss/FBt2/detection_loss‚0.7180043239493986 "loss/total_loss‚8.962895498434 % _timestamp‚1.7848466229918096e+09 2loss/SRqqt2/assignment_loss‚0.5397072732218791 /loss/FBt1/detection_loss‚0.6011005001901476 trainer/global_step‚4049 /loss/FRt2/assignment_loss‚0.802693590755164 2loss/SRqqt1/assignment_loss‚0.5686213264608463 1loss/SRqqt2/detection_loss‚0.6319619710841179 epoch‚14 _runtime‚7386.349004357 2loss/FBt1/assignment_loss‚0.031216822534521203 /loss/FRt2/detection_loss‚1.5046267008713095 1loss/SRqqt1/detection_loss‚1.0112948972920655MF°“ ÷„ _runtime‚7386.349004357 2loss/FBt1/assignment_loss‚0.031216822534521203 /loss/FRt2/detection_loss‚1.5046267008713095 1loss/SRqqt1/detection_loss‚1.0112948972920655 _step‚337 0loss/FRt1/assignment_loss‚1.9112284068623417 1loss/FBt2/assignment_loss‚0.04800093995527983 /loss/FRt1/detection_loss‚0.5944387452569273 /loss/FBt2/detection_loss‚0.7180043239493986 "loss/total_loss‚8.962895498434 % _timestamp‚1.7848466229918096e+09 2loss/SRqqt2/assignment_loss‚0.5397072732218791 /loss/FBt1/detection_loss‚0.6011005001901476 trainer/global_step‚4049 /loss/FRt2/assignment_loss‚0.802693590755164 2loss/SRqqt1/assignment_loss‚0.5686213264608463 1loss/SRqqt2/detection_loss‚0.6319619710841179 epoch‚14Ñ$´¯žø‚(:“ §²ŠÓÆϵ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.929' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.97633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Y)ÉEù‚(:º  §²ŠÓ´¿Õ³ disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚25429.40283203125 disk.drbd1584.in‚0$ memory_percent‚92.08731651306152 disk.drbd1285.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚25429.40283203125 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1944.62109375 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 194.34765625 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2883.0078125 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 70.68359375* proc.memory.percent‚11.730988820393879 cpu‚25.310470723115237$ disk./.usageGB‚1260.851764678955 disk.drbd1938.out‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.51343117871444 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 34.974609375 disk.sda2.out‚ 194.34765625 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 network.sent‚ 7.235804e+06 network.recv‚435770 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0ʾ#Ržú‚(:“ ¶²ŠÓó•Ø³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚284.074' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.69133333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“_¼ôEû‚(:º  ¶²ŠÓǽ÷³) disk./.usagePercent‚70.51341688507911 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.730988820393879 disk.drbd1965.in‚0 disk.drbd1289.out‚0 network.sent‚ 7.295845e+06 disk.sda.out‚ 194.63671875 disk.drbd2078.out‚0 disk.drbd1584.in‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚25429.59814453125 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1356.out‚0% disk./.usageGB‚1260.8515090942383 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 network.recv‚437478 disk.drbd1817.in‚0& disk.nvme0n1.out‚25429.59814453125 disk.drbd1226.out‚0 disk.drbd1510.in‚0 cpu‚25.31041414093848 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 194.63671875 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1673.in‚0) proc.memory.availableMB‚ 1944.62109375$ memory_percent‚92.08731651306152 disk.sda1.out‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 35.037109375 disk.nvme0n1.in‚ 70.68359375 disk.drbd1552.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1552.out‚0‡F¬Àœü‚(:‘ ŲŠÓö–̳ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚193.358' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.45266666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú¬"Dý‚(:¹  ŲŠÓëòë³ disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.dm-0.in‚ 70.68359375 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.rbd1.out‚ 35.037109375 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 cpu‚25.310360322583612 network.recv‚438470 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.730988820393879 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8516235351562 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚25430.09912109375$ memory_percent‚92.08731651306152 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51342328521433 disk.sda.in‚0.03125 disk.sda.out‚ 195.0859375 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2883.0078125 network.sent‚ 7.296965e+06 disk.drbd1673.out‚0 disk.sda2.out‚ 195.0859375) proc.memory.availableMB‚ 1944.62109375# disk.dm-0.out‚25430.09912109375 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0ý<Aþ‚(:¶  Ô²ŠÓÓÜݳ disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.51342477857922 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 cpu‚25.310239963436583 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd2111.in‚0$ disk.nvme0n1.out‚25430.373046875 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1356.out‚0$ memory_percent‚92.08731651306152 disk.sda2.in‚0.03125! disk.dm-0.out‚25430.373046875) proc.memory.availableMB‚ 1944.62109375 disk.sda.out‚ 195.40234375 disk.sda2.out‚ 195.40234375 disk.drbd2111.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1673.in‚0 network.sent‚ 7.29995e+06 network.recv‚439556 disk.drbd1584.out‚0 disk.sda1.out‚0$ disk./.usageGB‚1260.851650238037 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 35.0693359375 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2883.0078125* proc.memory.percent‚11.730988820393879[‡=êŸÿ‚(:” Ô²ŠÓž€Ä³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.889' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.96300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 æ—íB€‚(:·  㲊ÓͰݳ disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0% disk./.usageGB‚1260.8515930175781 disk.drbd1285.in‚0( disk./.usagePercent‚70.5134215785116% disk.nvme0n1.out‚25431.1572265625 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1817.in‚0 disk.sda2.out‚ 195.73828125 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 35.3427734375* proc.memory.percent‚11.730988820393879 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.sda1.in‚0" disk.dm-0.out‚25431.1572265625 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 195.73828125 disk.drbd2111.out‚0 network.sent‚ 7.304281e+06 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0$ memory_percent‚92.08733240763345 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 cpu‚25.31015270857163 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0( proc.memory.availableMB‚ 1944.6171875 network.recv‚439730 disk.drbd1931.out‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1828.out‚0 disk.rbd1.in‚0iö[{‘‚(:† 㲊ÓÀŒÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚185.34' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚61.78 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Ôm ž‚‚(:“ ò²ŠÓ±Þ³ gpu.0.gpu‚72.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.334' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.44466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LCçAƒ‚(:¶  ò²ŠÓ¾Âý³ disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 196.08203125 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 196.08203125( disk./.usagePercent‚70.5134205118224 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 70.68359375$ disk.nvme0n1.out‚25431.681640625 disk.dm-0.in‚ 70.68359375 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 network.recv‚440664" proc.memory.rssMB‚ 2883.0078125 cpu‚25.31006399130314 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8515739440918 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125* proc.memory.percent‚11.730988820393879 disk.drbd1356.out‚0 disk.drbd1510.in‚0$ memory_percent‚92.08633104960123 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1944.86328125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 35.4052734375 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0! disk.dm-0.out‚25431.681640625 disk.drbd1673.in‚0 disk.drbd1289.out‚0 network.sent‚ 7.307862e+06­') 2„- _wandbruntime‚7471  _runtime‚7471àëÌÆ…À & lr-AdamW/pg1‚9.612795239472676e-05 & lr-AdamW/pg2‚9.612795239472676e-05 trainer/global_step‚4099 $ _timestamp‚1.784846708337567e+09 _runtime‚7471.694886515 _step‚3389ôˆŠ& lr-AdamW/pg1‚9.612795239472676e-05 & lr-AdamW/pg2‚9.612795239472676e-05 trainer/global_step‚4099 $ _timestamp‚1.784846708337567e+09 _runtime‚7471.694886515 _step‚338ÒÙP°G2‡- _wandbruntime‚7471  _runtime‚7471‚ÇÖˆýO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12468772 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4276990940160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12468429 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323081288713216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1183741268648448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1183741268648448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35561162 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35561162 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4276990940160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1179465649317888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35561156 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23393968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚7471.696286787 _step‚339 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328308932212224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38057124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38047060 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1179465623144448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1183268542008779 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5227692668928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4268485688731 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1179000081748016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328308932212224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38057124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38047060 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1179465623144448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1183268542008779 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5227692668928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4268485688731 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1179000081748016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323081288713216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1183741268648448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1183741268648448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35561162 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35561162 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4276990940160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1179465649317888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35561156 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23393968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚7471.696286787 _step‚339 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5228938321920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4270851101769 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5228887062016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066220 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1179744532434944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4278013455360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _step‚340 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1184022572063744 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1184022572063744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73635749 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1183549734623561 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12471347 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1179278883521792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328388762141184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1179744558608384 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73625665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328389181492224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12471687 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35569523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38066220 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23399178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1183548364584195 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38056142 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35569529 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1184021200446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10927831 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4269506490371 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚4099 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824힤i ŒŒN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328388762141184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1179744558608384 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73625665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328389181492224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12471687 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35569523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38066220 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23399178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1183548364584195 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38056142 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35569529 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 T?DeviceStatsMonitor.o´'è©n_train_batch_end/allocated_bytes.all.freed‚1184021200446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10927831 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4269506490371 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚4099 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1179744532434944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10927836 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323159875079168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7473.175705202 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5228938321920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4270851101769 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5228887062016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066220 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1179744532434944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4278013455360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _step‚340 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1184022572063744 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1184022572063744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73635749 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1183549734623561 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12471347 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1179278883521792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824ÔrV2- _wandbruntime‚7473  _runtime‚7473ÞŠ,gŽü .loss/FRt1/assignment_loss‚1.91583641440055 ,loss/FRt1/detection_loss‚0.5412267362562 /loss/FBt2/detection_loss‚0.6107911351719937 %loss/total_loss‚9.347074270497094 2loss/SRqqt2/assignment_loss‚0.5734416692214128 2loss/FBt2/assignment_loss‚0.049468711254214766 1loss/SRqqt1/detection_loss‚1.1084415591006775 1loss/SRqqt2/detection_loss‚0.6384929021564959 _runtime‚7473.210517086 0loss/FRt2/assignment_loss‚0.8905385723378924 1loss/SRqqt1/assignment_loss‚0.629177207574849 epoch‚14 2loss/FBt1/assignment_loss‚0.028425570291476795 .loss/FRt2/detection_loss‚1.474429868935711 /loss/FBt1/detection_loss‚0.8868039237956206 trainer/global_step‚4099 $ _timestamp‚1.784846709853305e+09 _step‚341çmá $ _timestamp‚1.784846709853305e+09 _step‚341 .loss/FRt1/assignment_loss‚1.91583641440055 ,loss/FRt1/detection_loss‚0.5412267362562 /loss/FBt2/detection_loss‚0.6107911351719937 %loss/total_loss‚9.347074270497094 2loss/SRqqt2/assignment_loss‚0.5734416692214128 2loss/FBt2/assignment_loss‚0.049468711254214766 1loss/SRqqt1/detection_loss‚1.1084415591006775 1loss/SRqqt2/detection_loss‚0.6384929021564959 _runtime‚7473.210517086 0loss/FRt2/assignment_loss‚0.8905385723378924 1loss/SRqqt1/assignment_loss‚0.629177207574849 epoch‚14 2loss/FBt1/assignment_loss‚0.028425570291476795 .loss/FRt2/detection_loss‚1.474429868935711 /loss/FBt1/detection_loss‚0.8868039237956206 trainer/global_step‚4099Õƒ†äž‚(:“ ³ŠÓƒº¬³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.484' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.49466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›$;YD‘‚(:¹  ³ŠÓÐÈ̳ disk.sda.out‚ 196.75390625# disk.dm-0.out‚25432.51708984375 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 196.75390625 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 cpu‚25.30994686615513 network.sent‚ 7.368068e+06 disk.drbd1510.out‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1584.out‚0% disk./.usageGB‚1260.8559799194336 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd2111.out‚0* proc.memory.percent‚11.730988820393879 network.recv‚442485 disk.drbd2118.out‚0 disk.drbd1817.out‚0' proc.memory.availableMB‚ 1944.546875 disk.drbd1965.out‚0) disk./.usagePercent‚70.51366691702859 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0$ memory_percent‚92.08761850992838 disk.drbd1289.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚25432.51708984375 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 35.4052734375 disk.drbd1552.in‚0°‚Ù>’‚(:³  ³ŠÓ¨üþ³ disk.drbd2041.out‚0 disk.drbd1931.out‚0 network.sent‚ 7.368134e+06 disk.drbd1938.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2883.0078125* proc.memory.percent‚11.730988820393879 disk.drbd1289.out‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.sda2.in‚0.03125 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0$ memory_percent‚92.08761850992838' proc.memory.availableMB‚ 1944.546875 network.recv‚442551 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0$ disk./.usageGB‚1260.856502532959& disk.nvme0n1.out‚25432.74755859375 disk.drbd1673.in‚0 disk.sda2.out‚ 197.09765625 disk.nvme0n1.in‚ 70.68359375 disk.drbd1510.out‚0 disk.drbd1552.in‚0 cpu‚25.309924286411825 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚35.4375 disk.drbd1965.out‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51369614431277 disk.dm-0.in‚ 70.68359375 disk.rbd1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.sda.out‚ 197.09765625 disk.drbd1673.out‚0# disk.dm-0.out‚25432.74755859375 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0´÷³Ÿ“‚(:” ³ŠÓ“¬Þ³ gpu.0.gpu‚73.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.547' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.84900000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì㔂(:… Ÿ³ŠÓ£ôܳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.0' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û‹š–9•‚(:®  Ÿ³ŠÓ™“÷³ disk.drbd1931.in‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.730988820393879 disk.drbd2078.out‚0 disk.drbd1828.out‚0 cpu‚25.30983875667246 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd2041.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 70.68359375' proc.memory.availableMB‚ 1944.546875 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.drbd1938.out‚0$ memory_percent‚92.08761850992838 disk.drbd2111.in‚0$ disk.nvme0n1.out‚25433.435546875 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 197.390625 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 network.recv‚442659 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 197.390625 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0" proc.memory.rssMB‚ 2883.0078125) disk./.usagePercent‚70.51369187755596! disk.dm-0.out‚25433.435546875 disk.drbd1356.out‚0 disk.drbd1938.in‚0 network.sent‚ 7.368242e+06% disk./.usageGB‚1260.8564262390137 disk.drbd2111.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 35.5390625 disk.sda1.out‚03 Úž–‚(:“ ®³ŠÓàóѳ gpu.0.gpu‚87.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.437' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.14566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eQ F:—‚(:¯  ®³ŠÓÐÉð³ disk.drbd1938.out‚0) disk./.usagePercent‚70.51368569075858 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 197.609375 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0.03125 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.730988820393879 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.dm-0.in‚ 70.68359375" proc.memory.rssMB‚ 2883.0078125 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 network.sent‚ 7.36866e+06 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0$ disk./.usageGB‚1260.856315612793 disk.rbd1.out‚ 35.6015625 disk.drbd1552.in‚0 disk.drbd2118.in‚0 network.recv‚443606 disk.drbd1828.out‚0$ disk.nvme0n1.out‚25433.787109375 disk.drbd1817.out‚0 disk.sda2.out‚ 197.609375 disk.drbd1289.out‚0 cpu‚25.309786560011453 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0! disk.dm-0.out‚25433.787109375 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0$ memory_percent‚92.08763440450033) proc.memory.availableMB‚ 1944.54296875 disk.drbd1550.out‚0 disk.drbd1938.in‚0ÕÞuûž˜‚(:“ ½³ŠÓǺ¼³ gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.548' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.18266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)V]@™‚(:µ  ½³ŠÓù¦Ù³ disk.sda.out‚ 198.08984375 disk.drbd1673.in‚0 disk.dm-0.in‚ 70.68359375 disk.drbd1828.in‚0% disk.nvme0n1.out‚25434.7568359375 disk.drbd1550.out‚0$ memory_percent‚92.08765029907227 cpu‚25.30966951184264" disk.dm-0.out‚25434.7568359375 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51368825081266 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1944.5390625 disk.drbd1289.out‚0 disk.rbd1.out‚ 35.6015625 network.sent‚ 7.376894e+06% disk./.usageGB‚1260.8563613891602 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2078.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.730988820393879 network.recv‚444770 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 198.08984375 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17~@;à2š- _wandbruntime‚7558  _runtime‚7558¹´ÑÆ›À & lr-AdamW/pg1‚9.561108949724229e-05 & lr-AdamW/pg2‚9.561108949724229e-05 trainer/global_step‚4149 $ _timestamp‚1.784846795508891e+09 _runtime‚7558.866114562 _step‚3424_†Ò˜Štrainer/global_step‚4149 $ _timestamp‚1.784846795508891e+09 _runtime‚7558.866114562 _step‚342 & lr-AdamW/pg1‚9.561108949724229e-05 & lr-AdamW/pg2‚9.561108949724229e-05Ö…Qú2- _wandbruntime‚7558  _runtime‚7558#î©~(žýO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38513076 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1193411113842688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11057191 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚332301110403072 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1192940170436816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4328180496384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1197739294339072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5290086131200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5290135301120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1192940145008848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38503012 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 _step‚343 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23672134 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1197737922729472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12614943 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38503012 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35979506 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4326835060224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332300693142016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚7558.867464077 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327010607010816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1193411113842688 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23672134 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1197737922729472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12614943 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38503012 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35979506 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4326835060224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332300693142016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚7558.867464077 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327010607010816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1193411113842688 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5290135301120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1192940145008848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38503012 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005×ÈÛì5ž ‚(:“ ̳ŠÓքس gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚267.032' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.01066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é¾…B¡‚(:·  ̳ŠÓ¨Ïø³ disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0$ memory_percent‚92.08761850992838* proc.memory.percent‚11.730988820393879 network.sent‚ 7.379942e+06 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 70.68359375 disk.drbd1550.out‚0 disk.sda.out‚ 198.42578125) disk./.usagePercent‚70.51368505074505 disk.drbd1552.out‚0 disk.sda2.out‚ 198.42578125 disk.drbd1931.in‚0" disk.dm-0.out‚25435.0927734375 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125' proc.memory.availableMB‚ 1944.546875% disk./.usageGB‚1260.8563041687012 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 network.recv‚445896 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 cpu‚25.30958289012427 disk.rbd1.out‚ 35.7822265625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 70.68359375 disk.drbd2111.in‚0" proc.memory.rssMB‚ 2883.0078125 disk.drbd1828.out‚0 disk.drbd1965.in‚0% disk.nvme0n1.out‚25435.0927734375À^ð2¢- _wandbruntime‚7560  _runtime‚7560Ü:“î '£†N XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4320610373179 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74499967 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332380524143104 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35987879 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1193689996959744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23677712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4327857568256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1193690023133184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚7560.348360218 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35987873 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5291382026240 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4321954984577 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74510051 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38512094 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1193218946782624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1198019226144768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74510051 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1193689996959744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1197540927195169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4327857568256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12618226 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5291330766336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1198017854528000 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74499967 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38522172 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23677712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4327857568256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1193690023133184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚7560.348360218 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35987873 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5291382026240 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4321954984577 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74510051 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38512094 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1193218946782624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1198019226144768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74510051 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1193689996959744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1197540927195169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4327857568256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12618226 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5291330766336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1198017854528000 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74499967 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38522172 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38969028 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75356820 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1206880259125616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4370693454347 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336292464414208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1207356578367488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336292887966720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4372038065633 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚347 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36397856 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4378024616448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11189196 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1211735948420096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1211735948420096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38958964 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1206880233697648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11189191 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36397856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12761271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38958964 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330939925308416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1211734576810496 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38969028 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75356820 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1206880259125616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4370693454347 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336292464414208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1207356578367488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336292887966720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4372038065633 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚347 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36397856 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4378024616448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11189196 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1211735948420096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1211735948420096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38958964 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1206880233697648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11189191 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36397856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12761271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38958964 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330939925308416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1211734576810496 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36406229 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36406223 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23955991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336372300757504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11191836 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75384353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1207159035471424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4373058867385 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1212015880225792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38978124 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5353838245888 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚4199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38968046 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1211530749727411 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.cË÷bÉfurrent‚ 417253888 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331018511674368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5353789083136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _runtime‚7647.361656848 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38968046 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1207635487657984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75374269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4371714255987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4380392567808 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1207635487657984 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1207635461484544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36406229 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336372718011392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824»ñt'ºŒN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11191836 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75384353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1207159035471424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4373058867385 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1212015880225792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38978124 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5353838245888 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚4199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38968046 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1211530749727411 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331018511674368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5353789083136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _runtime‚7647.361656848 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38968046 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1207635487657984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75374269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4371714255987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4380392567808 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1207635487657984 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1207635461484544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36406229 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336372718011392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331018879765504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36406229 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36406223 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23955991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336372300757504Ü ²Ô2»- _wandbruntime‚7647  _runtime‚7647ü`ð¼€ 1loss/FBt2/assignment_loss‚0.08081572689242339 1loss/SRqqt1/detection_loss‚0.9241243574927798 /loss/FBt2/detection_loss‚0.5487754050182638 trainer/global_step‚4199 0loss/FRt2/assignment_loss‚0.8428246565163136 epoch‚14 _runtime‚7647.393628124 2loss/SRqqt1/assignment_loss‚0.5561085631273436 2loss/SRqqt2/assignment_loss‚0.5450111121199578 1loss/SRqqt2/detection_loss‚0.6659733190454842 #loss/total_loss‚9.1534709545634 _step‚349 1loss/FBt1/assignment_loss‚0.08711579475635839 /loss/FRt1/detection_loss‚0.5807624436335035 /loss/FRt2/detection_loss‚1.5798530599388938 /loss/FBt1/detection_loss‚0.9068836402148008 % _timestamp‚1.7848468840364757e+09 0loss/FRt1/assignment_loss‚1.8352228758072762~MœÁ ½… /loss/FBt2/detection_loss‚0.5487754050182638 trainer/global_step‚4199 0loss/FRt2/assignment_loss‚0.8428246565163136 epoch‚14 _runtime‚7647.393628124 2loss/SRqqt1/assignment_loss‚0.5561085631273436 2loss/SRqqt2/assignment_loss‚0.5450111121199578 1loss/SRqqt2/detection_loss‚0.6659733190454842 #loss/total_loss‚9.1534709545634 _step‚349 1loss/FBt1/assignment_loss‚0.08711579475635839 /loss/FRt1/detection_loss‚0.5807624436335035 /loss/FRt2/detection_loss‚1.5798530599388938 /loss/FBt1/detection_loss‚0.9068836402148008 % _timestamp‚1.7848468840364757e+09 0loss/FRt1/assignment_loss‚1.8352228758072762 1loss/FBt2/assignment_loss‚0.08081572689242339 1loss/SRqqt1/detection_loss‚0.9241243574927798Ýdß,?¾‚(:´  ¦´ŠÓ‘Øå³ disk.sda1.out‚0 disk.drbd1289.in‚0 disk.sda.in‚0.03125 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0" proc.memory.rssMB‚ 2883.0078125 proc.cpu.threads‚17$ memory_percent‚92.09009806315103 disk.drbd1289.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 200.7734375* proc.memory.percent‚11.730988820393879 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 36.314453125 disk.drbd1285.in‚0) disk./.usagePercent‚70.51337656422719 disk.drbd1552.in‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚25436.89892578125$ disk./.usageGB‚1260.850788116455 disk.sda2.in‚0.03125 disk.dm-0.in‚ 70.68359375 disk.drbd1510.in‚0% proc.memory.availableMB‚ 1943.9375 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.sda.out‚ 200.7734375 disk.drbd2041.out‚0 network.sent‚ 7.451309e+06 disk.drbd1817.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 70.68359375 cpu‚25.309959719622306# disk.dm-0.out‚25436.89892578125 network.recv‚449724 disk.drbd1584.in‚01Fž¿‚(:“ ¦´ŠÓîƒÄ³ gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.157' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.05233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J›ABÀ‚(:·  µ´ŠÓâ™Ù³ network.recv‚450700 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1931.in‚0" disk.dm-0.out‚25437.2900390625# proc.memory.rssMB‚ 2891.08984375 proc.cpu.threads‚17) disk./.usagePercent‚70.51334861697003 disk.sda.out‚ 201.0390625 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 70.68359375 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.sda1.in‚0* proc.memory.percent‚11.763874689737957 disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2111.out‚0 cpu‚25.314454742220892 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 70.68359375 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0( proc.memory.availableMB‚ 1935.9296875 disk.sda2.out‚ 201.0390625 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0% disk.nvme0n1.out‚25437.2900390625 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 36.494140625 disk.drbd1285.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 network.sent‚ 7.456956e+06% disk./.usageGB‚1260.8502883911133 disk.drbd1965.out‚0 disk.drbd1510.out‚0$ memory_percent‚92.12268193562826ÇiIŸÁ‚(:” µ´ŠÓêõ¹³ gpu.0.gpu‚95.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.115' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.03833333333334 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s.BC2Â- _wandbruntime‚7666  _runtime‚7666c-û*éw 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023121748119592667 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8700873255729675 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5599092245101929 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )jet/accuracy_2_of_3‚0.555472731590271 'particle/accuracy‚0.849185049533844 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5698059797286987 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8252896070480347 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28699207305908203 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6558934450149536 *jet/accuracy_1_of_4‚0.9707711338996887 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6568998694419861 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09579378366470337 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8269649147987366 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19451184570789337 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4476432502269745 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6868146061897278 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873339533805847 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21805334091186523 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25041550397872925 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19241134822368622 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23184645175933838 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028431134298443794 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42585229873657227 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2309478223323822 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1975998878479004 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.606431782245636 _runtime‚7666.787746235 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7162765860557556 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7659253478050232 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7136827707290649 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 epoch‚14 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472221493721008 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 particle/accuracy_1_of_1‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1746828407049179 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6427996754646301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4318182170391083 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9121355414390564 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947888255119324 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7444267868995667 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19309528172016144 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7682622671127319 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038068972527980804 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02571505308151245 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4565429091453552 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0372234508395195 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9964285492897034 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5974716544151306 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6778838038444519 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20542731881141663 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7719593048095703 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7576923370361328 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2333647757768631 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7384825348854065 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1597222238779068 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1597222238779068 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6632928252220154 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9982576370239258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7239459753036499 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.415794312953949 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9344692230224609 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11348936706781387 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 +particle/sensitivity‚0.6324610114097595 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6440318822860718 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1783325970172882 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5626713633537292 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6743808388710022 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.843441903591156 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8576388359069824 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47985759377479553 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9626067876815796 )loss/val_total_loss‚8.886683304369786 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21308067440986633 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8678907752037048 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7512801289558411 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025515317916870117 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 *jet/accuracy_1_of_2‚0.7852849364280701 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959821701049805 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8700873255729675 *jet/accuracy_3_of_4‚0.1030372604727745 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17887257039546967 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01920272968709469 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7222234010696411 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9844282269477844 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6762371063232422 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 % _timestamp‚1.7848469034296155e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21772336959838867 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841569662094116 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 *jet/accuracy_1_of_1‚0.3314363658428192 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3991709351539612 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7766753435134888 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09994713217020035 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99842369556427 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6951417922973633 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07599435746669769 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7validation_average_jet_accuracy‚0.37318557500839233 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7428211569786072 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6725571751594543 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7643289566040039 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7769914865493774 +jet/accuracy_2_of_2‚0.15450631082057953 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3993399143218994 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.613461554050441jU”§56 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7180880904197693 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950546622276306 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7905483841896057 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7882035374641418 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20035041868686676 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 +particle/specificity‚0.9083895087242126 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7889686822891235 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1805555671453476 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6778838038444519 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9983766674995422 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023121748119592667 *jet/accuracy_2_of_4‚0.8436643481254578 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.210552379488945 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.038919951766729355 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2142275720834732 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6611341834068298 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.982608437538147 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6672863960266113 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7658495306968689 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 trainer/global_step‚4199 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7907391786575317 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7124155759811401 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957275986671448 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16770581901073456 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9979591965675354 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08244901150465012 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982571601867676 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9049622416496277 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1805555671453476 +jet/accuracy_3_of_3‚0.06319533288478851 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6711612343788147 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7783575654029846 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1794585883617401 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21805334091186523 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988095223903656 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02878924272954464 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2803228497505188 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984253644943237 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6690316200256348 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6951041221618652 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04371893033385277 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _step‚350 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3227849304676056 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4142421782016754 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9985564351081848 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19866414368152618 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 particle/accuracy_3_of_3‚1¯¥IA´;Ä®w 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959821701049805 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962406158447266 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8700873255729675 *jet/accuracy_3_of_4‚0.1030372604727745 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17887257039546967 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01920272968709469 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7222234010696411 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9844282269477844 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6762371063232422 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 % _timestamp‚1.7848469034296155e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21772336959838867 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841569662094116 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 *jet/accuracy_1_of_1‚0.3314363658428192 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3991709351539612 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7766753435134888 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09994713217020035 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99842369556427 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6951417922973633 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07599435746669769 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7validation_average_jet_accuracy‚0.37318557500839233 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7428211569786072 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6725571751594543 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7643289566040039 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7769914865493774 +jet/accuracy_2_of_2‚0.15450631082057953 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3993399143218994 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6134615540504456 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7180880904197693 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950546622276306 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7905483841896057 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7882035374641418 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20035041868686676 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 +particle/specificity‚0.9083895087242126 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7889686822891235 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1805555671453476 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6778838038444519 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9983766674995422 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023121748119592667 *jet/accuracy_2_of_4‚0.8436643481254578 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.210552379488945 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.038919951766729355 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2142275720834732 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6611341834068298 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.982608437538147 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6672863960266113 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7658495306968689 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 trainer/global_step‚4199 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7907391786575317 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7124155759811401 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957275986671448 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16770581901073456 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9979591965675354 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08244901150465012 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982571601867676 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9049622416496277 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1805555671453476 +jet/accuracy_3_of_3‚0.06319533288478851 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6711612343788147 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7783575654029846 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1794585883617401 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21805334091186523 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988095223903656 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02878924272954464 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2803228497505188 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984253644943237 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6690316200256348 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6951041221618652 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04371893033385277 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _step‚350 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3227849304676056 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4142421782016754 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9985564351081848 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19866414368152618 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 particle/accuracy_3_of_3‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023121748119592667 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8700873255729675 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5599092245101929 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )jet/accuracy_2_of_3‚0.555472731590271 'particle/accuracy‚0.849185049533844 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5698059797286987 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8252896070480347 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28699207305908203 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6558934450149536 *jet/accuracy_1_of_4‚0.9707711338996887 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6568998694419861 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09579378366470337 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8269649147987366 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19451184570789337 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4476432502269745 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6868146061897278 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873339533805847 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21805334091186523 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25041550397872925 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19241134822368622 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23184645175933838 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028431134298443794 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42585229873657227 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2309478223323822 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1975998878479004 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.606431782245636 _runtime‚7666.787746235 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7162765860557556 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7659253478050232 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7136827707290649 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 epoch‚14 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472221493721008 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 particle/accuracy_1_of_1‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1746828407049179 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6427996754646301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4318182170391083 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9121355414390564 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947888255119324 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7444267868995667 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19309528172016144 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7682622671127319 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038068972527980804 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02571505308151245 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4565429091453552 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0372234508395195 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9964285492897034 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5974716544151306 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6778838038444519 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20542731881141663 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7719593048095703 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7576923370361328 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2333647757768631 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7384825348854065 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1597222238779068 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1597222238779068 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6632928252220154 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9982576370239258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7239459753036499 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.415794312953949 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9344692230224609 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11348936706781387 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 +particle/sensitivity‚0.6324610114097595 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6440318822860718 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1783325970172882 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5626713633537292 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6743808388710022 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.843441903591156 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8576388359069824 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47985759377479553 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9626067876815796 )loss/val_total_loss‚8.886683304369786 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21308067440986633 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8678907752037048 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7512801289558411 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025515317916870117 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 *jet/accuracy_1_of_2‚0.7852849364280701ÞŒD¸žÅ‚(:“ Ä´ŠÓ€Ä³ gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚278.119' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.70633333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïýd6?Æ‚(:´  Ä´ŠÓ‚Ôܳ disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 disk.rbd1.out‚ 36.556640625 disk.nvme0n1.in‚71.53125) disk./.usagePercent‚70.51357432840565 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8543243408203 disk.drbd1552.out‚0" disk.dm-0.out‚26407.0322265625 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 2917.87890625 cpu‚25.313551349587055 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2892.41015625* proc.memory.percent‚11.769247055053711 disk.drbd1552.in‚0 disk.sda.out‚ 201.58203125 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.sda2.out‚ 201.58203125 disk.dm-0.in‚71.53125 network.sent‚ 7.457064e+06 network.recv‚450808 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0$ memory_percent‚88.12712033589682 disk.drbd2078.in‚0% disk.nvme0n1.out‚26407.0322265625 disk.drbd1226.in‚0¶YžÇ‚(:“ Ó´ŠÓ‹¢×³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚267.445' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.14833333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c+ú:È‚(:¯  Ó´ŠÓ¾£÷³% disk.nvme0n1.out‚26407.3095703125 disk.drbd1356.out‚0 disk.nvme0n1.in‚71.53125 disk.drbd1550.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2892.41015625 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 36.658203125" disk.dm-0.out‚26407.3095703125 disk.drbd1550.out‚0 disk.sda.in‚0.03125) disk./.usagePercent‚70.51357432840565 disk.drbd1931.in‚0 disk.drbd2041.out‚0& proc.memory.availableMB‚ 2917.28125 network.sent‚ 7.517229e+06 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 201.8515625 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0$ memory_percent‚88.12955220540366 cpu‚25.313491296165918 network.recv‚452760 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0% disk./.usageGB‚1260.8543243408203 disk.dm-0.in‚71.53125 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.769247055053711 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 201.8515625 disk.drbd1226.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0MX±žÉ‚(:“ â´ŠÓøǺ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚289.723' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.57433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù‚)Ã;Ê‚(:°  â´ŠÓŒøÞ³ disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 network.sent‚ 7.517295e+06 disk.sda2.out‚ 202.4296875 disk.drbd2078.in‚0" disk.dm-0.out‚27377.0205078125 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 202.4296875 disk.drbd1356.out‚0 disk.nvme0n1.in‚71.53125 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0* proc.memory.percent‚11.769247055053711 disk.drbd1584.in‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8505477905273 disk.drbd1289.in‚0 disk.dm-0.in‚71.53125 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 cpu‚25.31333731922284 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 36.720703125 disk.drbd1510.in‚0( proc.memory.availableMB‚ 2917.2421875 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2892.41015625 disk.sda1.in‚0 disk.sda2.in‚0.03125$ memory_percent‚88.12971115112305) disk./.usagePercent‚70.51336312394321 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0% disk.nvme0n1.out‚27377.0205078125 proc.cpu.threads‚17 network.recv‚452826 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0úÌ̞˂(:“ ñ´ŠÓÿý̳ gpu.0.gpu‚17.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚309.045' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.01500000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÆÍÁ=Ì‚(:²  ñ´ŠÓÕöñ³ disk.sda.out‚ 202.9765625 disk.drbd1289.out‚0 disk.rbd1.out‚ 36.720703125 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0( proc.memory.availableMB‚ 2917.1484375# proc.memory.rssMB‚ 2892.41015625) disk./.usagePercent‚70.51347128622852 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 cpu‚25.313310535234926 disk.drbd1931.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125$ disk./.usageGB‚1260.852481842041 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚71.53125 disk.drbd2111.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0$ memory_percent‚88.13009262084961 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.769247055053711 disk.drbd1550.out‚0 disk.drbd1584.out‚0 network.recv‚454408 disk.sda2.out‚ 202.9765625 network.sent‚ 7.574248e+06 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚71.53125& disk.nvme0n1.out‚27377.79443359375 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0# disk.dm-0.out‚27377.79443359375 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0gyÐZžÍ‚(:“ €µŠÓÌ¿é³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.363' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.78766666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q‡j>΂(:³  €µŠÓÒ»‡´$ disk./.usageGB‚1260.852481842041 disk.drbd2078.out‚0 disk.drbd1285.out‚0$ memory_percent‚88.13047409057617 disk.drbd1817.in‚0 disk.sda.out‚ 203.42578125 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚71.53125 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚71.53125 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.769247055053711 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 37.353515625 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0 network.recv‚454474 disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 2917.0546875 cpu‚25.31315504104415 disk.sda2.out‚ 203.42578125 disk.drbd1289.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2892.41015625 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51347128622852 disk.sda1.in‚0 network.sent‚ 7.574314e+06 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0# disk.dm-0.out‚27378.17822265625 disk.drbd2078.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚27378.17822265625 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0¼ñ•Ö?Ï‚(:´  µŠÓ¸«É³ disk.nvme0n1.in‚71.53125 disk.drbd1510.out‚0 disk.drbd1550.out‚0 network.recv‚455473 disk.drbd1552.in‚0 disk.drbd1510.in‚0 network.sent‚ 7.579907e+06 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51347021953931% disk./.usageGB‚1260.8524627685547 disk.sda2.in‚0.03125 disk.sda2.out‚ 203.70703125 disk.drbd1965.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.dm-0.in‚71.53125 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚27378.80126953125 disk.sda.out‚ 203.70703125 disk.drbd1828.in‚0# disk.dm-0.out‚27378.80126953125 disk.drbd2111.in‚0 cpu‚25.313100914503256 disk.sda1.out‚0 disk.drbd1289.out‚0' proc.memory.availableMB‚ 2917.046875# proc.memory.rssMB‚ 2892.41015625* proc.memory.percent‚11.769247055053711 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 38.080078125 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0$ memory_percent‚88.13050587972006 disk.drbd2078.out‚0 disk.drbd1356.in‚0®pß·žÐ‚(:“ µŠÓÀ篳 gpu.0.gpu‚95.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚280.571' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.52366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}QÝÅ2Ñ- _wandbruntime‚7757  _runtime‚7757ËrõÇÒÁ _step‚351 & lr-AdamW/pg1‚9.456286421488101e-05 & lr-AdamW/pg2‚9.456286421488101e-05 trainer/global_step‚4249 % _timestamp‚1.7848469943582346e+09 _runtime‚7757.715836495áøØæÌÓÆ & lr-AdamW/pg1‚9.456286421488101e-05 & lr-AdamW/pg2‚9.456286421488101e-05 trainer/global_step‚4249 % _timestamp‚1.7848469943582346e+09 _runtime‚7757.715836495 _step‚351ßáÈv2Ô- _wandbruntime‚7757  _runtime‚7757SÅ,¿ÕýO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1227806729714085 LDDeviceStatsMonitor.on_8ð×ü&train_batch_start/segment.small_pool.allocated‚824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340340894307840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36905638 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24266616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5422704552960 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1227808099753339 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4428750787755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36905644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12938533 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚4249 _step‚352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39462228 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1223862719617024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚7757.716455313 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1228298897665024 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1223379323537616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76367866 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1223862719617024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76377936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7848469943590653e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436178048000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11328088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11328083 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1228297526055424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1228297526055424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436178048000 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1223379348965584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚3403413157632007hã(Ö‚P TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36905644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12938533 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚4249 _step‚352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39462228 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1223862719617024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚7757.716455313 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1228298897665024 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1223379323537616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76367866 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1223862719617024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76377936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7848469943590653e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436178048000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11328088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11328083 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1228297526055424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1228297526055424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436178048000 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1223379348965584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340341315763200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1227806729714085 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340340894307840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36905638 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24266616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5422704552960 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1227808099753339 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4428750787755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0àõÙ¯®2×- _wandbruntime‚7759  _runtime‚7759"[G '؇N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5423900389376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 trainer/global_step‚4249 _runtime‚7759.190577867 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1223658125311392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1228578829470720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4435855119872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76385315 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚334996829485056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39481388 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39471310 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1228086552289501 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36914005 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1224141628907520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚338 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12941781 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437200563200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340420729874432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24272509 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5423955843584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1224141602734080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39471310 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24272166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1228087922328867 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36914011 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11330728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340421153419776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1224141628907520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1223658150739360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76385315 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36914011 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76395399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334997197576192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12941443 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4428426978109 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_Töƒ$xtrain_batch_end/inactive_split.small_pool.allocated‚12941781 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437200563200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340420729874432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24272509 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5423955843584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1224141602734080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39471310 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24272166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1228087922328867 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36914011 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11330728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340421153419776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1224141628907520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1223658150739360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76385315 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36914011 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76395399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334997197576192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12941443 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4428426978109 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5423900389376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 trainer/global_step‚4249 _runtime‚7759.190577867 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0áw-ãÐ2Ú- _wandbruntime‚7759  _runtime‚7759õEèÛÛ /loss/FRt1/assignment_loss‚1.873701644858076 1loss/SRqqt2/detection_loss‚0.6506888543179395 %loss/total_loss‚8.707566319349137 _step‚354 2loss/FBt1/assignment_loss‚0.008908242331946623 1loss/FBt2/assignment_loss‚0.06218656857364947 /loss/FRt1/detection_loss‚0.5839462810698475 /loss/FBt2/detection_loss‚0.5028613471299145 epoch‚15 /loss/FRt2/assignment_loss‚0.640796631092838 1loss/SRqqt1/detection_loss‚0.9837421729053611 /loss/FBt1/detection_loss‚0.6989903616960402 trainer/global_step‚4249 % _timestamp‚1.7848469958690937e+09 _runtime‚7759.226278715 2loss/SRqqt1/assignment_loss‚0.7405165143007264 2loss/SRqqt2/assignment_loss‚0.5241634581127425 /loss/FRt2/detection_loss‚1.4370642429600549è`> ܆ % _timestamp‚1.7848469958690937e+09 _runtime‚7759.226278715 2loss/SRqqt1/assignment_loss‚0.7405165143007264 2loss/SRqqt2/assignment_loss‚0.5241634581127425 /loss/FRt2/detection_loss‚1.4370642429600549 /loss/FRt1/assignment_loss‚1.873701644858076 1loss/SRqqt2/detection_loss‚0.6506888543179395 %loss/total_loss‚8.707566319349137 _step‚354 2loss/FBt1/assignment_loss‚0.008908242331946623 1loss/FBt2/assignment_loss‚0.06218656857364947 /loss/FRt1/detection_loss‚0.5839462810698475 /loss/FBt2/detection_loss‚0.5028613471299145 epoch‚15 /loss/FRt2/assignment_loss‚0.640796631092838 1loss/SRqqt1/detection_loss‚0.9837421729053611 /loss/FBt1/detection_loss‚0.6989903616960402 trainer/global_step‚4249∰ñÝ‚(:’ žµŠÓƒ«³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚338.645' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚112.88166666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÆ>Þ‚(:³  žµŠÓˆ¬Í³& disk.nvme0n1.out‚27379.09228515625 disk.drbd2111.in‚0 network.sent‚ 7.580057e+06 network.recv‚455623 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd2078.in‚0$ memory_percent‚88.13110987345377( proc.memory.availableMB‚ 2916.8984375 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.769247055053711 disk.drbd1828.out‚0 disk.rbd1.out‚ 38.322265625 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚71.53125# proc.memory.rssMB‚ 2892.41015625$ disk./.usageGB‚1260.852481842041 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 204.09765625 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.sda.out‚ 204.09765625# disk.dm-0.out‚27379.09228515625 disk.drbd1552.out‚0 cpu‚25.31304349639531 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚71.53125 disk.drbd1552.in‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51347128622852 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0d"Cgžß‚(:“ ­µŠÓ—¨ç³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.916' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.97200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£‘ÃŒ@à‚(:µ  ­µŠÓÞÌ…´ disk.drbd1226.in‚0 disk.sda.out‚ 204.46484375 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 204.46484375 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51347021953931 disk.rbd1.out‚ 38.322265625 disk.drbd1965.out‚0$ memory_percent‚88.13117345174155 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.dm-0.in‚71.53125# disk.dm-0.out‚27379.27978515625 disk.drbd2041.in‚0 disk.drbd1817.out‚0# proc.memory.rssMB‚ 2892.41015625 disk.drbd1584.in‚0 disk.drbd1931.in‚0 cpu‚25.312983505880805 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 network.sent‚ 7.585654e+06 network.recv‚456510% disk./.usageGB‚1260.8524627685547 disk.drbd1828.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚27379.27978515625 disk.drbd1226.out‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.769247055053711 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚71.53125 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0( proc.memory.availableMB‚ 2916.8828125 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1938.out‚0Ò1z—:á‚(:¯  ¼µŠÓòî‚´ disk.sda2.out‚ 204.984375 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 204.984375 cpu‚25.31295880423299 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2892.41015625 disk.sda2.in‚0.03125 disk.dm-0.in‚71.53125 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1938.in‚0$ memory_percent‚88.13117345174155 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0% disk.nvme0n1.out‚27379.9560546875 disk.drbd1673.in‚0 disk.drbd2078.out‚0% disk./.usageGB‚1260.8524627685547 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0( proc.memory.availableMB‚ 2916.8828125* proc.memory.percent‚11.769247055053711 network.sent‚ 7.585762e+06) disk./.usagePercent‚70.51347021953931 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.sda1.out‚0" disk.dm-0.out‚27379.9560546875 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 network.recv‚456618 disk.rbd1.out‚ 38.3544921875 disk.nvme0n1.in‚71.53125 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1584.in‚0 disk.drbd1817.out‚00ôW@žâ‚(:“ ¼µŠÓ‚°à³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚260.103' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.70100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±µ¹ã‚(:’ ˵ŠÓðس gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚111.88' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.29333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¦àu<ä‚(:±  ˵ŠÓè§ö³ disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda2.out‚ 205.265625 disk.drbd2078.in‚0% disk./.usageGB‚1260.8524627685547 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.dm-0.in‚71.53125 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.769247055053711 disk.drbd1510.out‚0 disk.sda.out‚ 205.265625 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0$ memory_percent‚88.13045819600424 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 network.recv‚457986) disk./.usagePercent‚70.51347021953931 disk.drbd1552.out‚0" disk.dm-0.out‚27380.2568359375 disk.sda2.in‚0.03125 cpu‚25.312870996694315 disk.drbd1552.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.rbd1.out‚ 38.4560546875 disk.drbd2078.out‚0 disk.nvme0n1.in‚71.53125# proc.memory.rssMB‚ 2892.41015625 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 network.sent‚ 7.648406e+06 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚27380.2568359375) proc.memory.availableMB‚ 2917.05859375 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1289.out‚0/®žå‚(:“ ÚµŠÓê‚«³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚278.246' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.74866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü\”7Fæ‚(:»  ÚµŠÓçžÏ³ disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 network.sent‚ 7.648472e+06) disk./.usagePercent‚70.51345101913364 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 38.5185546875 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 71.53515625 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 71.53515625# disk.dm-0.out‚27380.72021484375 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.rbd1.in‚0$ memory_percent‚88.13045819600424 network.recv‚458052% disk./.usageGB‚1260.8521194458008* proc.memory.percent‚11.769247055053711 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0) proc.memory.availableMB‚ 2917.05859375 proc.cpu.threads‚17 disk.sda.out‚ 205.6171875 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 205.6171875& disk.nvme0n1.out‚27380.72021484375# proc.memory.rssMB‚ 2892.41015625 cpu‚25.312848492174844 disk.drbd1584.in‚0,ùE2ç- _wandbruntime‚7844  _runtime‚7844Ù[ÇèÁ % _timestamp‚1.7848470810870428e+09 _runtime‚7844.444278749 _step‚355 & lr-AdamW/pg1‚9.403163922832267e-05 & lr-AdamW/pg2‚9.403163922832267e-05 trainer/global_step‚4299Ä"DùÌéÆ & lr-AdamW/pg1‚9.403163922832267e-05 & lr-AdamW/pg2‚9.403163922832267e-05 trainer/global_step‚4299 % _timestamp‚1.7848470810870428e+09 _runtime‚7844.444278749 _step‚355ãص;2ê- _wandbruntime‚7844  _runtime‚7844xŽ(ëüO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77252238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚4299 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37323988 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1242295551746048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338847929507840 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1241797922285693 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1242294180136448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1237808184141824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37323994 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1241799292324947 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24546185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1237808184141824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1237319437654384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1237319412226416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39928244 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848470810881429e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37323994 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13085749 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4479854670563 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344333290116096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39928244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11460088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487367604224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338847561416704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5485360608256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1237808157968384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39918180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487367604224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24545832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13086097 _runtime‚ 7844.44893328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11460083 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39918180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1237808157968384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37323988 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1242295551746048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338847929507840 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1241797922285693 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1242294180136448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1237808184141824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37323994 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1241799292324947 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceSVºŒ¦—tatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24546185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1237808184141824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1237319437654384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1237319412226416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39928244 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848470810881429e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37323994 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13085749 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4479854670563 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344333290116096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39928244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11460088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487367604224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338847561416704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5485360608256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1237808157968384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39918180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487367604224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24545832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13086097 _runtime‚ 7844.44893328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11460083 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39918180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1237808157968384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1242574111934976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5486608425984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4487044676096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344413124299776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37332355 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1237598239428160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1242575483551744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39927262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13088649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚337 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77269701 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39937340 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24551372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _step‚357 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488390119424 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77269701 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4487044676096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77259617 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 % _timestamp‚1.7848470825632358e+09 _runtime‚7845.920823208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77259617 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37332355 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1238087067258880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37332361 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39937340 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344412704948736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4480875472315 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1242077744861109 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1237598214000192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4479530860917 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1238087093432320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24551714 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11462728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1242079114900475 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0UE‹'ñŒN ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1238087067258880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37332361 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488390119424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5486557166080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77269701 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4487044676096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77259617 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 % _timestamp‚1.7848470825632358e+09 _runtime‚7845.920823208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77259617 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37332355 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1238087067258880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37332361 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39937340 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344412704948736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4480875472315 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1242077744861109 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1237598214000192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4479530860917 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1238087093432320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24551714 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11462728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1242079114900475 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1242574111934976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5486608425984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4487044676096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344413124299776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37332355 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1237598239428160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1242575483551744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39927262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13088649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚337 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77269701 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39937340 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24551372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _step‚357 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488390119424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5547703847424 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1251753648666624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24825025 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1251259526343184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1251753648666624 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1251753622493184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348324951652864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 $ _timestamp‚1.784847168129529e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24824678 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537211724288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342777247805440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1256290834217472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4529613942085 trainer/global_step‚4349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538557160448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7931.487063657 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1256290834217472 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1251753622493184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976’–L‰(„P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5547703847424 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1251753648666624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24825025 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1251259526343184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1251753648666624 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1251753622493184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348324951652864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 $ _timestamp‚1.784847168129529e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24824678 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537211724288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342777247805440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1256290834217472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4529613942085 trainer/global_step‚4349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538557160448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7931.487063657 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1256290834217472 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1251753622493184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538557160448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40384196 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11592088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600è>{ç2…- _wandbruntime‚7932  _runtime‚79321@Ðü¸ †‡N K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78144003 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24830581 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40393292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348404380948480 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4531979355123 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segmentŸ^—U.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37750711 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚361 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4539579675648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7848471696123908e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1256572137632768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1252032531783680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78133919 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5548914868224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1256070307472083 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 trainer/global_step‚4349 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342855466080256 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1256068937432717 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1251538302688992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78133919 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40383214 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40383214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11594728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1256570766016000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13235510 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7932.970049602 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348404798202368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5548964030976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78144003 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37750705 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37750711 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40393292 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342855834171392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538234232320 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24830233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13235853 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4538234232320Ým¤'‡ŒN aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5548964030976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78144003 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37750705 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37750711 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40393292 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342855834171392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538234232320 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24830233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13235853 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4538234232320 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78144003 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24830581 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40393292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348404380948480 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4531979355123 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37750711 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _step‚361 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4539579675648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7848471696123908e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1256572137632768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1252032531783680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78133919 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5548914868224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1256070307472083 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 trainer/global_step‚4349 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342855466080256 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1256068937432717 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1251538302688992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78133919 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40383214 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40383214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11594728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1256570766016000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13235510 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7932.970049602 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348404798202368él„Ëi2ˆ- _wandbruntime‚7933  _runtime‚7933†gYÖ ‰ƒ trainer/global_step‚4349 _step‚362 /loss/FBt2/detection_loss‚0.5600235796034384 2loss/SRqqt1/assignment_loss‚0.6121191357086673 2loss/FBt2/assignment_loss‚0.031330485888208214 /loss/FRt1/detection_loss‚0.5465714755814451 1loss/SRqqt1/detection_loss‚1.0024210830852098 1loss/SRqqt2/detection_loss‚0.6616020727649806 0loss/FRt2/assignment_loss‚0.7752430070147794 $ _timestamp‚1.784847169643773e+09 %loss/total_loss‚9.006034169009565 epoch‚15 2loss/SRqqt2/assignment_loss‚0.5228636888001441 2loss/FBt1/assignment_loss‚0.007739371293326783 /loss/FRt2/detection_loss‚1.5406857132911682 _runtime‚7933.000968412 0loss/FRt1/assignment_loss‚1.7979016482718009 /loss/FBt1/detection_loss‚0.9475329077063963~¸ Šˆ 0loss/FRt1/assignment_loss‚1.7979016482718009 /loss/FBt1/detection_loss‚0.9475329077063963 trainer/global_step‚4349 _step‚362 /loss/FBt2/detection_loss‚0.5600235796034384 2loss/SRqqt1/assignment_loss‚0.6121191357086673 2loss/FBt2/assignment_loss‚0.031330485888208214 /loss/FRt1/detection_loss‚0.5465714755814451 1loss/SRqqt1/detection_loss‚1.0024210830852098 1loss/SRqqt2/detection_loss‚0.6616020727649806 0loss/FRt2/assignment_loss‚0.7752430070147794 $ _timestamp‚1.784847169643773e+09 %loss/total_loss‚9.006034169009565 epoch‚15 2loss/SRqqt2/assignment_loss‚0.5228636888001441 2loss/FBt1/assignment_loss‚0.007739371293326783 /loss/FRt2/detection_loss‚1.5406857132911682 _runtime‚7933.000968412êµÐ¿ E‹‚(:º  öŠÓ¥†ê³ disk.drbd1931.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 cpu‚25.312269937089063 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 207.9453125 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 71.53515625 disk.drbd1584.out‚0* proc.memory.percent‚11.769247055053711 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚27383.35693359375 disk.drbd2041.in‚0 network.recv‚461477 disk.drbd1226.in‚0 disk.drbd1510.out‚0$ memory_percent‚92.07894007364908 network.sent‚ 7.717207e+06 disk.drbd1550.out‚0( proc.memory.availableMB‚ 1946.6796875% disk./.usageGB‚1260.8521423339844 disk.drbd1938.in‚0 disk.dm-0.in‚ 71.53515625 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2892.41015625 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.sda1.out‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51345229916069 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1828.out‚0 disk.rbd1.out‚ 39.0126953125 disk.drbd1356.out‚0 disk.sda.out‚ 207.9453125 disk.drbd1965.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚27383.35693359375 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 ! °‘Œ‚(:† öŠÓ›éȳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚269.04' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.68 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07E¶œŸ‚(:” Ò¶ŠÓ†Ô¹³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚309.625' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.20833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ~™÷FŽ‚(:»  Ò¶ŠÓßÛ׳ disk.drbd1817.in‚0$ memory_percent‚92.07797050476074 disk.rbd1.in‚0 cpu‚25.31218522158764 network.sent‚ 7.790166e+06 disk.sda.in‚0.03125 disk.drbd2111.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.769247055053711# disk.dm-0.out‚27383.71533203125 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 39.1611328125& disk.nvme0n1.out‚27383.71533203125 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1946.91796875 proc.cpu.threads‚17 network.recv‚463135 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0) disk./.usagePercent‚70.51345123247148 disk.sda.out‚ 208.29296875 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 71.53515625 disk.nvme0n1.in‚ 71.53515625 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2892.41015625$ disk./.usageGB‚1260.852123260498 disk.drbd1673.in‚0 disk.sda2.out‚ 208.29296875 disk.drbd1931.in‚0uþzÒž‚(:“ ᶊÓ髳 gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.435' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.81166666666667 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â¦$G‚(:¼  ᶊÓȇͳ disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda1.in‚0* proc.memory.percent‚11.769247055053711% disk./.usageGB‚1260.8520736694336 disk.drbd1931.in‚0( proc.memory.availableMB‚ 1946.9140625 disk.drbd1285.out‚0 network.recv‚463201 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 208.55859375 disk.drbd1938.in‚0 disk.dm-0.in‚ 71.53515625# disk.dm-0.out‚27383.89892578125 disk.drbd1584.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 208.55859375 disk.drbd1965.in‚0 disk.drbd2078.out‚0$ memory_percent‚92.07798639933267 disk.nvme0n1.in‚ 71.53515625 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 network.sent‚ 7.790232e+06 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1550.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 cpu‚25.312162932330345 proc.cpu.threads‚17) disk./.usagePercent‚70.51344845907956 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125# proc.memory.rssMB‚ 2892.41015625 disk.rbd1.out‚ 39.1611328125& disk.nvme0n1.out‚27383.89892578125f%¥¡B‘‚(:·  ð¶ŠÓû•Ô³% disk.nvme0n1.out‚27384.3037109375 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 208.8671875 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51344525901195 disk.dm-0.in‚ 71.53515625 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2892.41015625 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 cpu‚25.312139785272137 disk.drbd1584.out‚0# memory_percent‚92.0789877573649 disk.drbd1938.out‚0" disk.dm-0.out‚27384.3037109375 disk.nvme0n1.in‚ 71.53515625 disk.drbd1226.out‚0 network.sent‚ 7.855551e+06 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 39.193359375 disk.sda1.in‚0 disk.drbd1550.out‚0% disk./.usageGB‚1260.8520164489746 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.769247055053711 network.recv‚464995 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 208.8671875) proc.memory.availableMB‚ 1946.66796875Å*gý’‚(:’ ð¶ŠÓ¬Ò´³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.62' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.87333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢““‚(:ˆ ÿ¶ŠÓ·þ°³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.034' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÝèD”‚(:¹  ÿ¶ŠÓÉÝÕ³ disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0% disk.nvme0n1.out‚27384.4755859375 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 network.sent‚ 7.855617e+06) disk./.usagePercent‚70.51348771324226 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 71.53515625" disk.dm-0.out‚27384.4755859375 disk.drbd1356.out‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 1946.66796875 proc.cpu.threads‚17% disk./.usageGB‚1260.8527755737305 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda.out‚ 209.37109375 disk.drbd1673.in‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2892.41015625 disk.drbd1584.in‚0 disk.dm-0.in‚ 71.53515625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 39.474609375 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0# memory_percent‚92.0789877573649 disk.drbd1817.out‚0 disk.sda2.out‚ 209.37109375 disk.drbd1550.in‚0 cpu‚25.312117014509045 network.recv‚465061 disk.drbd1931.in‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.769247055053711Uýš•‚(: Ž·ŠÓªôÖ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚234.4' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.13333333333333 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®Ô ÖF–‚(:»  Ž·ŠÓšøï³ disk.drbd1965.in‚0 cpu‚25.312061984298005 disk.sda.out‚ 209.58203125 disk.drbd1356.in‚0 proc.cpu.threads‚17 network.sent‚ 7.855725e+06% disk./.usageGB‚1260.8527717590332 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 71.53515625# memory_percent‚92.0789877573649) proc.memory.availableMB‚ 1946.66796875 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.769247055053711 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0) disk./.usagePercent‚70.51348749990443 disk.sda.in‚0.03125# disk.dm-0.out‚27385.07080078125 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 209.58203125 disk.drbd1965.out‚0 disk.drbd1510.in‚0 network.recv‚465169 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 71.53515625 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2892.41015625& disk.nvme0n1.out‚27385.07080078125 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 39.537109375 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125jgr¶2—- _wandbruntime‚8018  _runtime‚8018s©]ǘÁ & lr-AdamW/pg1‚9.295531271244736e-05 & lr-AdamW/pg2‚9.295531271244736e-05 trainer/global_step‚4399 % _timestamp‚1.7848472552484796e+09 _runtime‚8018.605739113 _step‚363£Pñ›Ì™Æ % _timestamp‚1.7848472552484796e+09 _runtime‚8018.605739113 _step‚363 & lr-AdamW/pg1‚9.295531271244736e-05 & lr-AdamW/pg2‚9.295531271244736e-05 trainer/global_step‚4399ëè“>È2š- _wandbruntime‚8018  _runtime‚8018úÄ'h‰›ûO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40840148 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13380207 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4582062436179 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40830084 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25104295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4589746716672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 _step‚364 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1265699113191424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13379852 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352316785198080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588401280512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _runtime‚ 8018.60710016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4580717824893 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1265699113191424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1265699087017984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 $ _timestamp‚1.784847255249555e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1265199615031984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38160694 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1270287488298496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1265699087017984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38160694 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346706566103040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1270288859908096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4588401280512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11724088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1269781677468163 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4589746716672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1270288859908096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5610219095040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25103935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 trainer/global_step‚4399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5610163633664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11724083œ¹ ž(œ€P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 trainer/global_step‚4399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5610163633664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11724083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40840148 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13380207 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4582062436179 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40830084 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25104295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4589746716672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 _step‚364 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1265699113191424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13379852 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352316785198080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588401280512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _runtime‚ 8018.60710016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4580717824893 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1265699113191424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1265699087017984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 $ _timestamp‚1.784847255249555e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1265199615031984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38160694 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1270287488298496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1265699087017984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38160694 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346706566103040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1270288859908096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4588401280512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11724088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1269781677468163 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4589746716672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1270288859908096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5610219095040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25103935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5ì³Ýg2- _wandbruntime‚8020  _runtime‚80201Ú¶) 'ž‡N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11726728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1265978022481920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4589423788544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11726723 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38169055 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1265977996308480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13383153 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79008221 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38169055 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352396203977216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352396623328256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1270061500043691 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40849244 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1265478416805760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1265478391377792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40839166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25109529 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5611470859264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚8020.083343532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38169061 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79018305 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚346785152468992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4583083237931 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38169061 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13382806 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864DvÌÛ ŸŒN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79008221 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38169055 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352396203977216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352396623328256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1270061500043691 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40849244 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1265478416805760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1265478391377792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40839166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25109529 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5611470859264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚8020.083343532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38169061 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79018305 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚346785152468992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4583083237931 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38169061 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13382806 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11726728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1265978022481920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4589423788544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11726723 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38169055 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1265977996308480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13383153í÷Kš2 - _wandbruntime‚8020  _runtime‚8020Äóí˜ ¡ƒ /loss/FRt1/detection_loss‚0.5636538138251536 0loss/FRt1/assignment_loss‚1.9359103604121968 1loss/SRqqt1/detection_loss‚1.0078242075606738 /loss/FBt2/detection_loss‚0.6516935257161339 %loss/total_loss‚9.107307522403563 _runtime‚8020.119896808 _step‚366 2loss/SRqqt1/assignment_loss‚0.6718466022571619 2loss/SRqqt2/assignment_loss‚0.5723617156524029 /loss/FRt2/detection_loss‚1.5104131530466918 1loss/SRqqt2/detection_loss‚0.6576973283624747 /loss/FBt1/detection_loss‚0.7230350887950729 epoch‚15 trainer/global_step‚4399 0loss/FBt2/assignment_loss‚0.0104701890290394 % _timestamp‚1.7848472567626731e+09 0loss/FRt2/assignment_loss‚0.8007476190188983 3loss/FBt1/assignment_loss‚0.0016539187276614468%Ô~¢ˆ 0loss/FRt2/assignment_loss‚0.8007476190188983 3loss/FBt1/assignment_loss‚0.0016539187276614468 /loss/FRt1/detection_loss‚0.5636538138251536 0loss/FRt1/assignment_loss‚1.9359103604121968 1loss/SRqqt1/detection_loss‚1.0078242075606738 /loss/FBt2/detection_loss‚0.6516935257161339 %loss/total_loss‚9.107307522403563 _runtime‚8020.119896808 _step‚366 2loss/SRqqt1/assignment_loss‚0.6718466022571619 2loss/SRqqt2/assignment_loss‚0.5723617156524029 /loss/FRt2/detection_loss‚1.5104131530466918 1loss/SRqqt2/detection_loss‚0.6576973283624747 /loss/FBt1/detection_loss‚0.7230350887950729 epoch‚15 trainer/global_step‚4399 0loss/FBt2/assignment_loss‚0.0104701890290394 % _timestamp‚1.7848472567626731e+09îû¥Ã…ž£‚(:“ ·ŠÓžÌس gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚284.507' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.83566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âBb·D¤‚(:¹  ·ŠÓ©îõ³ disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51351203375611 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1226.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2892.41015625 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0$ memory_percent‚92.07948048909506 network.sent‚ 7.861386e+06 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0& disk.nvme0n1.out‚27385.28173828125* proc.memory.percent‚11.769247055053711 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.rbd1.in‚0 cpu‚25.31194620233851 disk.drbd1931.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚27385.28173828125 disk.drbd2041.out‚0 disk.rbd1.out‚ 39.716796875 network.recv‚466139% disk./.usageGB‚1260.8532104492188 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 71.53515625 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 210.40234375 disk.drbd1828.out‚0' proc.memory.availableMB‚ 1946.546875 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 210.40234375 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 71.53515625 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0°Í¹!ž¥‚(:“ ¬·ŠÓÑܳ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚296.345' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.78166666666668 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÇB¦‚(:·  ¬·ŠÓ´«´ disk.drbd2111.in‚0 disk.dm-0.in‚ 71.53515625 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0$ memory_percent‚92.07960764567056 disk.sda.in‚0.03125 disk.sda2.out‚ 210.7265625 disk.drbd2041.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚27386.38916015625 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.769247055053711# proc.memory.rssMB‚ 2892.41015625) disk./.usagePercent‚70.51353187417531 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1550.out‚0 network.sent‚ 7.861494e+06 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 71.53515625 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 210.7265625 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1817.out‚0 network.recv‚466247% disk./.usageGB‚1260.8535652160645 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 39.779296875 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0' proc.memory.availableMB‚ 1946.515625 disk.drbd1289.in‚0 disk.drbd1510.in‚0 cpu‚25.31195436199266 disk.drbd1938.in‚0# disk.dm-0.out‚27386.38916015625¯C§‚(:¸  »·ŠÓ£°õ³ disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚27386.58837890625 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 71.53515625 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1946.01171875 disk.rbd1.out‚ 39.880859375# proc.memory.rssMB‚ 2892.41015625 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 211.1484375 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 network.recv‚467197 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0# disk.dm-0.out‚27386.58837890625 disk.sda.out‚ 211.1484375 disk.dm-0.in‚ 71.53515625 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚1260.853645324707 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 cpu‚25.311809160510276 disk.drbd1938.in‚0 disk.drbd1356.in‚0 network.sent‚ 7.86971e+06 disk.drbd2041.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51353635426997$ memory_percent‚92.08165804545084* proc.memory.percent‚11.769247055053711sv—¥ž¨‚(:“ »·ŠÓŠ—ѳ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.031' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.34366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôát8ž©‚(:“ Ê·ŠÓ­ܳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.666' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.88866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ#ý€Eª‚(:º  Ê·ŠÓçùü³ disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0" disk.dm-0.out‚27387.7822265625 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 40.208984375 disk.drbd1550.out‚0 network.recv‚467263 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 cpu‚25.311756067917138 disk.sda.out‚ 211.55078125# proc.memory.rssMB‚ 2892.41015625 disk.sda2.out‚ 211.55078125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.769247055053711 disk.drbd1550.in‚0 network.sent‚ 7.869776e+06 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 71.53515625% disk./.usageGB‚1260.8536071777344 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0$ memory_percent‚92.08197593688965 disk.dm-0.in‚ 71.53515625% disk.nvme0n1.out‚27387.7822265625 proc.cpu.threads‚17) proc.memory.availableMB‚ 1945.93359375) disk./.usagePercent‚70.51353422089157 disk.drbd1552.in‚0 disk.drbd1931.out‚0psøŸ«‚(:” Ù·ŠÓ°›§³ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.802' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.26733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hc?¬‚(:´  Ù·ŠÓô¦Æ³ disk.drbd1584.in‚0 disk.dm-0.in‚ 71.53515625 disk.drbd1965.in‚0# memory_percent‚92.0823891957601 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.769247055053711 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 network.sent‚ 7.869926e+06 network.recv‚467413 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 212.015625 disk.drbd1584.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 cpu‚25.311675584170455 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 40.208984375$ disk./.usageGB‚1260.853530883789 disk.drbd2041.in‚0) disk./.usagePercent‚70.51352995413475 disk.drbd2118.out‚0" disk.dm-0.out‚27388.3447265625 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 212.015625 disk.drbd1828.out‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1945.83203125# proc.memory.rssMB‚ 2892.41015625 disk.nvme0n1.in‚ 71.53515625% disk.nvme0n1.out‚27388.3447265625 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0ahøéB­‚(:·  è·ŠÓ¸”ã³ disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0$ memory_percent‚92.08242098490396 cpu‚25.311560326480183 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1289.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 network.sent‚ 7.932609e+06 disk.sda2.out‚ 212.3828125 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2892.41015625! disk.dm-0.out‚27389.169921875 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 79.92578125* proc.memory.percent‚11.769247055053711 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1945.82421875 proc.cpu.threads‚17% disk./.usageGB‚1260.8535118103027 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0$ disk.nvme0n1.out‚27389.169921875 disk.drbd1285.in‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51352888744556 disk.sda.out‚ 212.3828125 disk.dm-0.in‚ 79.92578125 disk.drbd1356.out‚0 network.recv‚469134 disk.rbd1.out‚ 40.2412109375 O)ž®‚(:“ è·ŠÓ¯¸¿³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.196' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.06533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š®ï=2¯- _wandbruntime‚8105  _runtime‚8105¢‚tœÇ°Á & lr-AdamW/pg1‚9.241035226487678e-05 & lr-AdamW/pg2‚9.241035226487678e-05 trainer/global_step‚4449 % _timestamp‚1.7848473421071022e+09 _runtime‚8105.464830237 _step‚367GÂí–Ì±Æ & lr-AdamW/pg1‚9.241035226487678e-05 & lr-AdamW/pg2‚9.241035226487678e-05 trainer/global_step‚4449 % _timestamp‚1.7848473421071022e+09 _runtime‚8105.464830237 _step‚367ïu†Û2²- _wandbruntime‚8105  _runtime‚8105‹&‡Š(³üO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279644577716224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11856088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1279139703720784 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1279644551542784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639590836736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4639590836736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350635884400640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1284284142379520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356308598563328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41286036 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38579038 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41296100 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4640936272896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13527643 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79865074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38579044 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25383385 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5672714162688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1283771500000517 trainer/global_step‚4449 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005ëƒç(î´P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639590836736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4639590836736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350635884400640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1284284142379520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356308598563328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41286036 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38579038 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41296100 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4640936272896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13527643 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79865074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38579044 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25383385 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5672714162688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1283771500000517 trainer/global_step‚4449 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279644577716224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11856088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1279139703720784 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1279644551542784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13530580 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1279418505494560 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1279923460833280 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _runtime‚8106.942064958 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79892607 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25388971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1284051322575933 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1279923487006720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38587405 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚337 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38587405 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350714102675456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11858723 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41295118 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1284565445794816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350714470766592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4641958788096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4641958788096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680¦H„'·ŒN YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79892607 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25388971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1284051322575933 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1279923487006720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38587405 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚337 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38587405 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350714102675456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11858723 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41295118 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1284565445794816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350714470766592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4641958788096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4641958788096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79882523 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41305196 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13530580 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1279418505494560 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1279923460833280 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _runtime‚8106.942064958ñ|¯š‰2¸- _wandbruntime‚8106  _runtime‚8106ÃE̹ 0loss/FRt1/assignment_loss‚1.8032752947725412 2loss/SRqqt1/assignment_loss‚0.5870046492484554 /loss/FRt2/detection_loss‚1.4051326977120222 /loss/FBt2/detection_loss‚0.6435944561651626 trainer/global_step‚4449 2loss/SRqqt2/assignment_loss‚0.5899597721831674 1loss/SRqqt1/detection_loss‚0.9935632851916207 %loss/total_loss‚8.726900543876276 epoch‚15 _step‚370 2loss/FBt1/assignment_loss‚0.006902176902550143 1loss/FBt2/assignment_loss‚0.02962037262317339 /loss/FRt1/detection_loss‚0.5178112303570562 1loss/SRqqt2/detection_loss‚0.6311926865871084 .loss/FBt1/detection_loss‚0.745175353131227 $ _timestamp‚1.784847343620463e+09 0loss/FRt2/assignment_loss‚0.7736685690021914 _runtime‚8106.977702287‘Ä º† .loss/FBt1/detection_loss‚0.745175353131227 $ _timestamp‚1.784847343620463e+09 0loss/FRt2/assignment_loss‚0.7736685690021914 _runtime‚8106.977702287 0loss/FRt1/assignment_loss‚1.8032752947725412 2loss/SRqqt1/assignment_loss‚0.5870046492484554 /loss/FRt2/detection_loss‚1.4051326977120222 /loss/FBt2/detection_loss‚0.6435944561651626 trainer/global_step‚4449 2loss/SRqqt2/assignment_loss‚0.5899597721831674 1loss/SRqqt1/detection_loss‚0.9935632851916207 %loss/total_loss‚8.726900543876276 epoch‚15 _step‚370 2loss/FBt1/assignment_loss‚0.006902176902550143 1loss/FBt2/assignment_loss‚0.02962037262317339 /loss/FRt1/detection_loss‚0.5178112303570562 1loss/SRqqt2/detection_loss‚0.6311926865871084òAˆDüž»‚(:“ ÷·ŠÓŠªÍ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.686' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.22866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓjÀY=¼‚(:²  ÷·ŠÓ’çí³ disk.sda2.out‚ 212.671875 disk.dm-0.in‚ 79.92578125 disk.rbd1.out‚ 40.3427734375 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 79.92578125 disk.drbd1226.out‚0 disk.drbd1510.out‚0# memory_percent‚92.0844554901123 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0! disk.dm-0.out‚27389.478515625 disk.drbd1965.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.769247055053711 disk.sda.out‚ 212.671875$ disk.nvme0n1.out‚27389.478515625 disk.drbd1226.in‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 1945.32421875 cpu‚25.311569192725795 disk.drbd1552.in‚0 network.sent‚ 7.94101e+06( disk./.usagePercent‚70.5135301674726% disk./.usageGB‚1260.8535346984863 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0 network.recv‚470303 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2892.41015625 disk.drbd1817.in‚0ü¤à蜽‚(:‘ †¸ŠÓÞ×­³ gpu.0.gpu‚75.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚107.6' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.86666666666666 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëø0Q¾‚(:¸  †¸ŠÓ‹˜Ï³ disk.drbd2118.in‚0" disk.dm-0.out‚27389.9326171875 disk.drbd1965.in‚0 disk.drbd1510.out‚0 dis15l¶k.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda2.in‚0.03125 disk.rbd1.out‚ 40.4052734375 disk.drbd1817.out‚0 disk.sda.out‚ 212.94140625 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2892.41015625 network.recv‚470411 disk.drbd1510.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0( disk./.usagePercent‚70.5135312341618 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 79.92578125 cpu‚25.311488237618548 disk.drbd1673.in‚0 disk.sda2.out‚ 212.94140625 disk.drbd1938.in‚0 disk.dm-0.in‚ 79.92578125 disk.drbd1550.out‚0' proc.memory.availableMB‚ 1945.796875 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0% disk.nvme0n1.out‚27389.9326171875 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0* proc.memory.percent‚11.769247055053711 network.sent‚ 7.941118e+06 disk.drbd1289.out‚0$ memory_percent‚92.08253224690756 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8535537719727 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1356.out‚0X– R“¿‚(:ˆ •¸ŠÓý¸²³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.916' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Nb<À‚(:±  •¸ŠÓâÉÕ³ disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.769247055053711 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1945.51171875! disk./.usageGB‚1260.853515625 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.dm-0.in‚ 80.203125 disk.drbd1289.out‚0 disk.drbd1938.out‚0$ memory_percent‚92.08369255065918 disk.drbd2078.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51352910078339 disk.drbd1965.out‚0% disk.nvme0n1.out‚27390.1591796875 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1285.in‚0 cpu‚25.311436282500775 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0" disk.dm-0.out‚27390.1591796875 disk.drbd1828.out‚0 disk.rbd1.out‚ 40.4052734375 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 80.203125 disk.sda2.out‚ 213.3359375# proc.memory.rssMB‚ 2892.41015625 network.recv‚470519 disk.drbd1356.out‚0 disk.drbd1226.in‚0 network.sent‚ 7.941226e+06 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda.out‚ 213.3359375ÚC öŸÁ‚(:” ¤¸ŠÓ…Öð³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚309.739' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.24633333333333 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®‚¹2‚(:§  ¤¸ŠÓ݈Œ´ disk.drbd2078.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.769247055053711) disk./.usagePercent‚70.51352696740499 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 80.203125 cpu‚25.311382078433287% disk./.usageGB‚1260.8534774780273 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0( proc.memory.availableMB‚ 1945.5078125 disk.drbd2041.in‚0 disk.drbd1828.out‚0# disk.nvme0n1.out‚27390.52734375 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.sda.out‚ 213.6171875 disk.dm-0.out‚27390.52734375 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0$ memory_percent‚92.08370844523111 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2892.41015625 disk.dm-0.in‚ 80.203125 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.rbd1.out‚40.5 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 network.recv‚470519 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 network.sent‚ 7.941226e+06 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 213.6171875 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.sda.in‚0.03125äî¢WœÃ‚(:‘ ³¸ŠÓàÔȳ gpu.0.gpu‚19.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚106.48' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.49333333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 «Rû4Ä‚(:©  ³¸ŠÓˆÚë³ disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.dm-0.out‚27390.69140625 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.829026540120442 disk.nvme0n1.in‚ 80.203125 disk.drbd1550.out‚0 disk.sda.in‚0.03125 network.recv‚471652 disk.dm-0.in‚ 80.203125 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 40.6015625 disk.drbd1817.out‚0 disk.drbd1938.out‚0 cpu‚25.315900825542638) disk./.usagePercent‚70.51352696740499$ memory_percent‚92.14197794596353% proc.memory.availableMB‚ 1931.1875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 network.sent‚ 7.949634e+06 disk.drbd1552.in‚0 disk.sda2.out‚ 214.1171875% disk./.usageGB‚1260.8534774780273 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0" proc.memory.rssMB‚ 2907.1015625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.sda2.in‚0.03125 disk.sda.out‚ 214.1171875 disk.drbd1965.out‚0# disk.nvme0n1.out‚27390.69140625ð4íR2Å- _wandbruntime‚8178  _runtime‚8178 ¡ú­;Ƨw 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7093127369880676 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.056641895323991776 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6247757077217102 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6488722562789917 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7643677592277527 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17370103299617767 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982066750526428 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868001341819763 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6929202675819397 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6440253257751465 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9985119104385376 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7430066466331482 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8044372200965881 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2608497440814972 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6807765364646912 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11277151107788086 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2083333283662796 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4283229410648346 *jet/accuracy_2_of_3‚0.5047113299369812 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7272727489471436 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19808411598205566 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5647379755973816 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1525624543428421 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5732843279838562 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03815443068742752 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7272727489471436 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6766436696052551 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7625773549079895 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6307691931724548 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9661188721656799 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3793656527996063 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974387884140015 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9834704995155334 trainer/global_step‚4479 )loss/val_total_loss‚8.642977897156149 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11332731693983078 (jet/accuracy_1_of_3‚0.91302090883255 +particle/specificity‚0.9032095074653625 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9209145307540894 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1846451759338379 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8030843734741211 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7978795170783997 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009894856251776218 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972015023231506 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4267549514770508 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20265702903270721 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8958333134651184 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039164431393146515 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18196354806423187 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04307541251182556 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7654433846473694 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7574998736381531 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977743029594421 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6523882746696472 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14129577577114105 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8822013139724731 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.040852662175893784 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8386470675468445 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9949018359184265 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8943021893501282 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1846451759338379 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16323460638523102 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_2_of_4‚0.8115915060043335 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7123647332191467 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1754877120256424 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6731433868408203 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8294758200645447 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6300155520439148 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19694963097572327 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15999208390712738 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29651927947998047 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977743029594421 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9933584332466125 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7576923370361328 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7327075004577637 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 *jet/accuracy_1_of_1‚0.3081829249858856 +particle/sensitivity‚0.6240707635879517 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3498670756816864 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 /particle/accuracy_4_of_4‚0.9598763585090637 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 (particle/accuracy‚0.8433197736740112 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6819912195205688 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7160783410072327 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43053165078163147 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6724982261657715 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6990436911582947 +jet/accuracy_3_of_3‚0.06697653979063034 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933093190193176 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6479583978652954 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23507849872112274 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019752709195017815 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03815443068742752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 epoch‚15 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889172911643982 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6335693597793579 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857297539710999 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7"Purity/2FRt0SRqqt1FBt/event_purity‚0.54358971118927 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12612974643707275 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41866761445999146 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7095687985420227 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028472721576690674 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983848333358765 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.995231568813324 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7658687233924866 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2963370084762573 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18537141382694244 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21818675100803375 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18245495855808258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6455163955688477 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6461240649223328 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18582311272621155 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19885994493961334 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17778106033802032 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8037777543067932 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_3_of_4‚0.10292945802211761 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9953916668891907 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14221559464931488 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6308788657188416 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5735200047492981 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6729785203933716 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1007031723856926 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _runtime‚8178.544598519 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 % _timestamp‚1.7848474151864684e+09 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9867168068885803 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7150247693061829 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2083333283662796 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19959750771522522 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18880963325500488 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8294758200645447 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8029993772506714 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6984300017356873 _step‚371 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1604159027338028 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7077160477638245 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6650566458702087 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9961414337158203 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45643308758735657 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37713298201560974 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7115563750267029 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42112234234809875 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7286339402198792 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6461240649223328 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04381096363067627 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.041300661861896515 7validation_average_jet_accuracy‚0.35069704055786133 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6882274746894836 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.620132565498352 /particle/accuracy_3_of_4‚0.9955357313156128 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8518933653831482 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24918602406978607 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0323210209608078 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0?ê3êµ&Ǭw & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14129577577114105 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8822013139724731 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.040852662175893784 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8386470675468445 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9949018359184265 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8943021893501282 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1846451759338379 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16323460638523102 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_2_of_4‚0.8115915060043335 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7123647332191467 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1754877120256424 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6731433868408203 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8294758200645447 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6300155520439148 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19694963097572327 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15999208390712738 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29651927947998047 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977743029594421 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9933584332466125 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7576923370361328 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7327075004577637 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 *jet/accuracy_1_of_1‚0.3081829249858856 +particle/sensitivity‚0.6240707635879517 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3498670756816864 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 /particle/accuracy_4_of_4‚0.9598763585090637 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 (particle/accuracy‚0.8433197736740112 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6819912195205688 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7160783410072327 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43053165078163147 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6724982261657715 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6990436911582947 +jet/accuracy_3_of_3‚0.06697653979063034 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933093190193176 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6479583978652954 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23507849872112274 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019752709195017815 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03815443068742752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 epoch‚15 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889172911643982 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6335693597793579 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857297539710999 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7"Purity/2FRt0SRqqt1FBt/event_purity‚0.54358971118927 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12612974643707275 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41866761445999146 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7095687985420227 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028472721576690674 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983848333358765 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.995231568813324 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7658687233924866 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2963370084762573 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18537141382694244 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21818675100803375 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18245495855808258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6455163955688477 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6461240649223328 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18582311272621155 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19885994493961334 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17778106033802032 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8037777543067932 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_3_of_4‚0.10292945802211761 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9953916668891907 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14221559464931488 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6308788657188416 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5735200047492981 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6729785203933716 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1007031723856926 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _runtime‚8178.544598519 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 % _timestamp‚1.7848474151864684e+09 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9867168068885803 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7150247693061829 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2083333283662796 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19959750771522522 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18880963325500488 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8294758200645447 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8029993772506714 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6984300017356873 _step‚371 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1604159027338028 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7077160477638245 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6650566458702087 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9961414337158203 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45643308758735657 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37713298201560974 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7115563750267029 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42112234234809875 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7286339402198792 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6461240649223328 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04381096363067627 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.041300661861896515 7validation_average_jet_accuracy‚0.35069704055786133 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6882274746894836 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.620132565498352 /particle/accuracy_3_of_4‚0.9955357313156128 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8518933653831482 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24918602406978607 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0323210209608078 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7093127369880676 >&Purity”%è?ý/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9615384340286255 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.056641895323991776 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6247757077217102 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6488722562789917 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7643677592277527 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17370103299617767 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982066750526428 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868001341819763 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6929202675819397 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6440253257751465 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9985119104385376 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7430066466331482 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8044372200965881 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2608497440814972 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6807765364646912 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11277151107788086 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2083333283662796 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4283229410648346 *jet/accuracy_2_of_3‚0.5047113299369812 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7272727489471436 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19808411598205566 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5647379755973816 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1525624543428421 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5732843279838562 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03815443068742752 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7272727489471436 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6766436696052551 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7625773549079895 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6307691931724548 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9661188721656799 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3793656527996063 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974387884140015 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9834704995155334 trainer/global_step‚4479 )loss/val_total_loss‚8.642977897156149 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11332731693983078 (jet/accuracy_1_of_3‚0.91302090883255 +particle/specificity‚0.9032095074653625 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9209145307540894 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1846451759338379 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8030843734741211 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7978795170783997 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009894856251776218 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972015023231506 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4267549514770508 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20265702903270721 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8958333134651184 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039164431393146515 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18196354806423187 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04307541251182556 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7654433846473694 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7574998736381531 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977743029594421 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6523882746696472ó©…èK”È‚(:‰ ¸ŠÓâ¶Á³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚308.433' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.811 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¹98É‚(:­  ¸ŠÓè™ã³ network.recv‚471718 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 cpu‚25.317179463107248 disk.drbd1584.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51352483402657& disk.nvme0n1.out‚27391.06103515625 disk.sda.out‚ 214.40625 disk.drbd2118.out‚0 disk.drbd1938.out‚0$ memory_percent‚92.14304288228354 disk.sda2.out‚ 214.40625" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 7.9497e+06 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 40.6640625 disk.nvme0n1.in‚ 80.203125 disk.dm-0.in‚ 80.203125 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1930.92578125% disk./.usageGB‚1260.8534393310547# disk.dm-0.out‚27391.06103515625Õ‘œ;=Ê‚(:²  ѸŠÓ´˜û³$ memory_percent‚92.14305877685547 disk.drbd1226.in‚0 cpu‚25.31711664183693 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0' proc.memory.availableMB‚ 1930.921875# disk.dm-0.out‚27391.26416015625 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 80.203125 disk.drbd1965.out‚0 network.recv‚473504 disk.sda2.in‚0.03125% disk./.usageGB‚1260.8534202575684 disk.rbd1.out‚ 40.6640625 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.sda.in‚0.03125 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 214.74609375 disk.drbd2041.in‚0) disk./.usagePercent‚70.51352376733738 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 214.74609375 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚27391.26416015625 disk.drbd2078.out‚0 disk.drbd1828.out‚0 network.sent‚ 8.009867e+06 disk.drbd1289.out‚0 disk.drbd2078.in‚0¢Š"‘Ë‚(:† ѸŠÓ·¼Ý³ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚284.19' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.73 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ŽËP2Ì- _wandbruntime‚8212  _runtime‚8212 AlçÆÍÀ & lr-AdamW/pg1‚9.186095540303112e-05 & lr-AdamW/pg2‚9.186095540303112e-05 trainer/global_step‚4499 % _timestamp‚1.7848474486542444e+09 _runtime‚ 8212.01149393 _step‚3727KËÎÅ _step‚372 & lr-AdamW/pg1‚9.186095540303112e-05 & lr-AdamW/pg2‚9.186095540303112e-05 trainer/global_step‚4499 % _timestamp‚1.7848474486542444e+09 _runtime‚ 8212.01149393ôðùR2Ï- _wandbruntime‚8212  _runtime‚8212DÏ(ÐýO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4696398829568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39086820 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38684160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1296150718965760 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80876115 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1300847091621888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11994975 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39086820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4688534429818 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360356182885888 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354613834120192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1296150692792320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80886185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚373 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41799359 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚8212.015934249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41789295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13700963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41789295 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5742348765696 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚4499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4696398829568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1296150692792320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚360356589661184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39086826 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5742387449856 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1300328672601856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406775296 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1300848463231488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11994980 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25695938 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1295638793560752 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1295638768132784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354614202211328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050Sùü–(Ñ‚P VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4696398829568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39086820 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38684160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1296150718965760 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80876115 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1300847091621888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11994975 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39086820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4688534429818 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360356182885888 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354613834120192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1296150692792320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80886185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚373 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41799359 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚8212.015934249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41789295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13700963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41789295 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5742348765696 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚4499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4696398829568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1296150692792320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚360356589661184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39086826 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5742387449856 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1300328672601856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406775296 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1300848463231488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11994980 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25695938 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1295638793560752 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1295638768132784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354614202211328õßÚÁî2Ò- _wandbruntime‚8213  _runtime‚8213¦u:ZÜÓ‡N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39095193 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 408865280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5743666802688 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1296429628256256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354692420486144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1296429628256256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1296429602082816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13703961 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚360436455379968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚40774144 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1300608495177384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4690899842856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80903648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4698766780928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41808455 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.lä7"1all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80893564 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11997615 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4698766780928 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80903648 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _runtime‚8213.490142384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80893564 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354692788577280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41808455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4697421337600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 trainer/global_step‚4499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1301127023420416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25701576 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39095193 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13704260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41798377 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11997620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41798377 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚360436046514688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1295917595334528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848474501325545e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39095187 _step‚374 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80903648 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _runtime‚8213.490142384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80893564 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354692788577280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41808455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4697421337600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 trainer/global_step‚4499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1301127023420416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25701576 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39095193 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13704260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41798377 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11997620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41798377 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚360436046514688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1295917595334528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848474501325545e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39095187 _step‚374 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4698766780928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41808455 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80893564 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11997615ö$«ô2Õ- _wandbruntime‚8213  _runtime‚8213Ze—ˆ Öƒ /loss/FBt1/detection_loss‚0.8753324826061726 _step‚375 1loss/FBt2/assignment_loss‚0.04679064654538073 trainer/global_step‚4499 % _timestamp‚1.7848474501672816e+09 _runtime‚8213.524480825 2loss/SRqqt1/assignment_loss‚0.6359948758892263 /loss/FRt2/detection_loss‚1.4316544023727291 epoch‚16 /loss/FBt2/detection_loss‚0.6875150478799049 %loss/total_loss‚9.105242453053492 0loss/FRt2/assignment_loss‚0.7731640589291646 2loss/SRqqt2/assignment_loss‚0.5485305903262645 2loss/FBt1/assignment_loss‚0.008493452989632146 /loss/FRt1/detection_loss‚0.5758235688450323 1loss/SRqqt1/detection_loss‚1.0043326628473153 1loss/SRqqt2/detection_loss‚0.6366591663448754 0loss/FRt1/assignment_loss‚1.8809514974777946օ׈ %loss/total_loss‚9.105242453053492 0loss/FRt2/assignment_loss‚0.7731640589291646 2loss/SRqqt2/assignment_loss‚0.5485305903262645 2loss/FBt1/assignment_loss‚0.008493452989632146 /loss/FRt1/detection_loss‚0.5758235688450323 1loss/SRqqt1/detection_loss‚1.0043326628473153 1loss/SRqqt2/detection_loss‚0.6366591663448754 0loss/FRt1/assignment_loss‚1.8809514974777946 /loss/FBt1/detection_loss‚0.8753324826061726 _step‚375 1loss/FBt2/assignment_loss‚0.04679064654538073 trainer/global_step‚4499 % _timestamp‚1.7848474501672816e+09 _runtime‚8213.524480825 2loss/SRqqt1/assignment_loss‚0.6359948758892263 /loss/FRt2/detection_loss‚1.4316544023727291 epoch‚16 /loss/FBt2/detection_loss‚0.6875150478799049÷ƒ=Ì¡žØ‚(:“ ชÓ©¦å³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚286.677' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.55900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä§GÓ9Ù‚(:®  ชÓ“Ü‚´ disk.drbd1828.out‚0 disk.sda.out‚ 214.953125' proc.memory.availableMB‚ 1930.890625 disk.drbd1673.in‚0" disk.dm-0.out‚27391.7470703125 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0" memory_percent‚92.143185933431 network.recv‚475411 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 40.6962890625 disk.drbd1226.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.sda2.out‚ 214.953125 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 network.sent‚ 8.076929e+06 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 80.203125) disk./.usagePercent‚70.51352483402657 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0% disk.nvme0n1.out‚27391.7470703125 disk.drbd1552.out‚0% disk./.usageGB‚1260.8534393310547 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1817.out‚0 cpu‚25.316994282997236 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 80.203125 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0Ìk/‘ŸÚ‚(:” ︊ÓÛÁ³³ gpu.0.gpu‚75.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.746' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.24866666666665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀU=Û‚(:²  ︊Ó„ÎÖ³* proc.memory.percent‚11.829026540120442 cpu‚25.316965646750543 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 215.42578125 disk.sda2.out‚ 215.42578125 disk.drbd1817.out‚0' proc.memory.availableMB‚ 1930.890625) disk./.usagePercent‚70.51352483402657 disk.dm-0.in‚ 80.203125 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚27391.9267578125 disk.drbd1510.in‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8534393310547" disk.dm-0.out‚27391.9267578125 disk.drbd1356.out‚0 disk.drbd1289.in‚0 network.recv‚475519 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0" memory_percent‚92.143185933431 disk.drbd2041.in‚0 disk.rbd1.out‚ 40.7978515625 disk.drbd1510.out‚0 proc.cpu.threads‚17 network.sent‚ 8.077037e+06 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.sda2.in‚0.03125T3è>Ü‚(:³  þ¸ŠÓ–Íú³ disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd1226.in‚0 disk.sda2.out‚ 215.7265625 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.sda.out‚ 215.7265625 proc.cpu.threads‚17) disk./.usagePercent‚70.51351182041829 disk.drbd1289.out‚0" disk.dm-0.out‚27395.3876953125 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚27395.3876953125 disk.drbd1226.out‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8532066345215 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0$ memory_percent‚92.14367866516113 network.sent‚ 8.080073e+06 disk.drbd1289.in‚0 disk.dm-0.in‚ 80.203125 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 network.recv‚476643 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.sda.in‚0.03125" proc.memory.rssMB‚ 2907.1015625 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 40.9462890625 disk.drbd1673.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1285.in‚0) proc.memory.availableMB‚ 1930.76953125 cpu‚25.31687275672106äÕì&žÝ‚(:“ þ¸ŠÓ¨õÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.063' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.35433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆrÚnžÞ‚(:“ ¹ŠÓÕãæ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.906' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.63533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cj¼&>ß‚(:³  ¹ŠÓåà†´ disk.drbd1828.in‚0# memory_percent‚92.1429475148519 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 cpu‚25.316781412233713 disk.drbd1289.out‚0 disk.drbd1356.out‚0" disk.dm-0.out‚27395.9697265625 disk.drbd1965.in‚0% disk.nvme0n1.out‚27395.9697265625 network.sent‚ 8.080223e+06 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.829026540120442) disk./.usagePercent‚70.51351182041829 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 network.recv‚476793 disk.sda.in‚0.03125 disk.sda2.out‚ 216.0078125 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1930.94921875% disk./.usageGB‚1260.8532066345215 disk.sda1.out‚0 disk.dm-0.in‚ 80.203125 disk.drbd2041.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 40.9462890625 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd1226.in‚0 disk.sda.out‚ 216.0078125dfˆ$Ÿà‚(:” œ¹ŠÓï¦Ó³ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚109.092' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.364000000000004 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý£W=á‚(:²  œ¹ŠÓé£ñ³% disk.nvme0n1.out‚27396.6416015625 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125" proc.memory.rssMB‚ 2907.1015625 disk.nvme0n1.in‚ 80.203125 disk.drbd1510.out‚0 disk.drbd1817.in‚0 cpu‚25.316782274636065 disk.sda2.in‚0.03125 disk.drbd2041.in‚0 network.sent‚ 8.14289e+06 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 216.30078125% disk./.usageGB‚1260.8532104492188 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0' proc.memory.availableMB‚ 1930.609375 disk.rbd1.out‚ 41.158203125 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 80.203125 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0$ memory_percent‚92.14433034261069 disk.drbd1673.out‚0 network.recv‚478538) disk./.usagePercent‚70.51351203375611 disk.sda.out‚ 216.30078125 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚27396.6416015625·ð‹Jžâ‚(:“ «¹ŠÓ¸“á³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚263.062' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.68733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ýï<ã‚(:±  «¹ŠÓܨ´ disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 216.6640625" disk.dm-0.out‚27396.9228515625 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd1226.out‚0* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1584.in‚0( proc.memory.availableMB‚ 1930.3046875 network.sent‚ 8.148526e+06 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 41.259765625% disk.nvme0n1.out‚27396.9228515625 disk.drbd1828.in‚0 cpu‚25.316661224263544 disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 80.203125 disk.drbd1356.in‚0 disk.drbd2111.out‚0$ disk./.usageGB‚1260.853229522705 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0$ memory_percent‚92.14557011922201 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.rbd1.in‚0 network.recv‚479480 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 216.6640625) disk./.usagePercent‚70.51351310044532 disk.drbd1931.in‚0" proc.memory.rssMB‚ 2907.1015625¯œ©S2ä- _wandbruntime‚8299  _runtime‚8299Jâ:ÆåÀ & lr-AdamW/pg2‚9.130719414025168e-05 trainer/global_step‚4549 $ _timestamp‚1.784847535657369e+09 _runtime‚8299.014671711 _step‚376 & lr-AdamW/pg1‚9.130719414025168e-05²²ËæÅ trainer/global_step‚4549 $ _timestamp‚1.784847535657369e+09 _runtime‚8299.014671711 _step‚376 & lr-AdamW/pg1‚9.130719414025168e-05 & lr-AdamW/pg2‚9.130719414025168e-05øÛë02ç- _wandbruntime‚8299  _runtime‚8299ûngÄèýO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5805345390080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748933821952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1314843745702912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358543520508928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1314845117312512 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747588385792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39505170 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13849853 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739638312626 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12126980 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42255311 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39505176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1310096183490560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4747588385792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42255311 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13850187 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚334 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7848475356585255e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42245247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364348497807872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81760487 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12126975 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1309578882249552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25977167 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81750417 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5805392462848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚4549 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1314318495134210 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25976828 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1314319865173464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358543152417792}K¿(é‚P F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚4549 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1314318495134210 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25976828 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1314319865173464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358543152417792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5805345390080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748933821952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1314843745702912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358543520508928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1314845117312512 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747588385792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39505170 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13849853 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739638312626 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12126980 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42255311 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39505176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1310096183490560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4747588385792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42255311 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13850187 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚334 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7848475356585255e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42245247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364348497807872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81760487 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12126975 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1309578882249552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25977167 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81750417 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5805392462848ù_u›2ê- _wandbruntime‚8300  _runtime‚8300]Èj 'ë‡N YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12129615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25982405 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5806649756672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4742003725664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1310375066607616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4748610893824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358621738783744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12129620 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39513537 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _step‚378 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39513543 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42254329 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚338 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1309857658595360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4740659114266 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1315125049118208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1315125049118208 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1310375092781056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364428756631552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1310375066607616 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1314599687748992 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1309857684023328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81777950 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81767866 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42254329 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749956337152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4748610893824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1315123677501440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749956337152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13853123 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39513537 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1314598317709626 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39513543 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13852790 _runtime‚8300.494119914 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5806602691072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚4549 % _timestamp‚1.7848475371365318e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1315125049118208 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1310375092781056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364428756 qð»631552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1310375066607616 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1314599687748992 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1309857684023328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81777950 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81767866 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42254329 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749956337152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4748610893824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1315123677501440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749956337152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13853123 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39513537 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1314598317709626 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39513543 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13852790 _runtime‚8300.494119914 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5806602691072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚4549 % _timestamp‚1.7848475371365318e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12129615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25982405 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5806649756672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4742003725664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1310375066607616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4748610893824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358621738783744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755ú¤´v2í- _wandbruntime‚8300  _runtime‚8300 Ÿ¯%î 0loss/FRt2/assignment_loss‚0.8400657708320001 2loss/SRqqt2/assignment_loss‚0.5242660000720202 1loss/FBt2/assignment_loss‚0.02777829800495863 1loss/SRqqt2/detection_loss‚0.6915920143481671 0loss/FRt1/assignment_loss‚1.8584759239919066 1loss/FBt1/assignment_loss‚0.02497504020825166 epoch‚16 % _timestamp‚1.7848475371737275e+09 _step‚379 2loss/SRqqt1/assignment_loss‚0.6759164237139955 .loss/FRt1/detection_loss‚0.568833727020937 /loss/FBt1/detection_loss‚0.7792032712957624 /loss/FBt2/detection_loss‚0.6412499567720479 %loss/total_loss‚9.059998674576413 /loss/FRt2/detection_loss‚1.4988261450503633 1loss/SRqqt1/detection_loss‚0.9288161032660028 trainer/global_step‚4549 _runtime‚8300.531069318åb¯† ï† /loss/FBt2/detection_loss‚0.6412499567720479 %loss/total_loss‚9.059998674576413 /loss/FRt2/detection_loss‚1.4988261450503633 1loss/SRqqt1/detection_loss‚0.9288161032660028 trainer/global_step‚4549 _runtime‚8300.531069318 0loss/FRt2/assignment_loss‚0.8400657708320001 2loss/SRqqt2/assignment_loss‚0.5242660000720202 1loss/FBt2/assignment_loss‚0.02777829800495863 1loss/SRqqt2/detection_loss‚0.6915920143481671 0loss/FRt1/assignment_loss‚1.8584759239919066 1loss/FBt1/assignment_loss‚0.02497504020825166 epoch‚16 % _timestamp‚1.7848475371737275e+09 _step‚379 2loss/SRqqt1/assignment_loss‚0.6759164237139955 .loss/FRt1/detection_loss‚0.568833727020937 /loss/FBt1/detection_loss‚0.7792032712957624ûc™žð‚(:“ º¹ŠÓßîµ³ gpu.0.gpu‚78.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.049' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.68299999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þq<ñ‚(:±  º¹ŠÓôٳ disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 80.203125 network.recv‚480301 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0% disk./.usageGB‚1260.8532104492188 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0) disk./.usagePercent‚70.51351203375611 disk.rbd1.out‚ 41.322265625$ memory_percent‚92.14590390523276) proc.memory.availableMB‚ 1930.22265625 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.sda.out‚ 216.9609375 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 216.9609375 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 network.sent‚ 8.151512e+06 disk.drbd1817.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 cpu‚25.316603269268096 disk.drbd1510.in‚0 disk.drbd1931.in‚0! disk.dm-0.out‚27397.302734375$ disk.nvme0n1.out‚27397.302734375 disk.drbd1356.in‚0ZÖ7¥žò‚(:“ ɹŠÓ‡Âdz gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚257.246' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.74866666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÞÊò8ó‚(:­  ɹŠÓÔ¨æ³ disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 41.501953125 disk.drbd1817.in‚0$ memory_percent‚92.14490254720053* proc.memory.percent‚11.829026540120442 disk.sda.out‚ 217.41796875& proc.memory.availableMB‚ 1930.46875% disk./.usageGB‚1260.8520050048828 disk.drbd1550.out‚0 disk.sda2.out‚ 217.41796875 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 cpu‚25.31648310940098 disk.drbd1285.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 network.sent‚ 8.15162e+06 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1673.out‚0" proc.memory.rssMB‚ 2907.1015625 network.recv‚480409 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd1226.out‚0 disk.drbd1550.in‚0! disk.dm-0.out‚27397.478515625 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0( disk./.usagePercent‚70.5134446189984$ disk.nvme0n1.out‚27397.478515625 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 80.203125 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0“oÀŸô‚(:” عŠÓ§ÝÚ³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.376' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.12533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g÷q>õ‚(:³  عŠÓ–«ù³ network.recv‚482085 cpu‚25.316452624094435 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd1673.in‚0 disk.sda.out‚ 217.69140625" proc.memory.rssMB‚ 2907.1015625 disk.drbd1226.in‚0 network.sent‚ 8.21167e+06 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0$ memory_percent‚92.14490254720053* proc.memory.percent‚11.829026540120442 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚27397.98486328125 disk.rbd1.out‚ 41.564453125 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 80.203125 disk.drbd2078.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51344355230921% disk./.usageGB‚1260.8519859313965 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125& proc.memory.availableMB‚ 1930.46875 disk.sda2.out‚ 217.69140625 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚27397.98486328125 disk.drbd1673.out‚0Ê¢8Š;ö‚(:°  繊Ó´à³ disk.drbd1673.out‚0 disk.rbd1.out‚ 41.666015625 disk.drbd1285.in‚0 disk.sda1.out‚0 network.sent‚ 8.211778e+06$ disk./.usageGB‚1260.852741241455 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 218.1328125 disk.drbd1931.in‚0 disk.dm-0.in‚ 80.203125 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0" proc.memory.rssMB‚ 2907.1015625 cpu‚25.316364854523673 disk.sda2.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1510.out‚0$ memory_percent‚92.14490254720053* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0( disk./.usagePercent‚70.5134857932017 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 80.203125 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 network.recv‚482193 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚27398.12548828125 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 218.1328125# disk.dm-0.out‚27398.12548828125 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0& proc.memory.availableMB‚ 1930.46875Ÿ¥ïŸ÷‚(:” 繊Óãóó gpu.0.gpu‚81.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.276' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.75866666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ˜dÆžø‚(:“ ö¹ŠÓŽöÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚291.841' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.28033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i­æ5ù‚(:ª  ö¹ŠÓ†íò³% disk./.usageGB‚1260.8526611328125 disk.dm-0.in‚ 80.203125 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1965.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0# disk.nvme0n1.out‚27398.75390625 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 network.sent‚ 8.223064e+06 disk.rbd1.out‚ 41.994140625 disk.drbd1226.in‚0 network.recv‚485033 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2907.1015625 cpu‚25.316275226903617 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0$ memory_percent‚92.14490254720053 proc.cpu.threads‚17 disk.sda2.out‚ 218.484375 disk.drbd1817.out‚0 disk.dm-0.out‚27398.75390625 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0& proc.memory.availableMB‚ 1930.46875 disk.drbd1584.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51348131310704 disk.sda.out‚ 218.484375 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 80.2031254ß+ÝŸú‚(:” …ºŠÓ‡ãâ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.629' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.87633333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÝSÍ8û‚(:­  …ºŠÓçùý³ disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0$ memory_percent‚92.14490254720053 disk.dm-0.in‚ 80.203125 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 network.sent‚ 8.223913e+06 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0# disk.nvme0n1.out‚27398.88671875 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.dm-0.out‚27398.88671875 network.recv‚485207 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda.out‚ 218.76171875" proc.memory.rssMB‚ 2907.1015625 cpu‚25.31618628877213 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.sda.in‚0.03125* proc.memory.percent‚11.829026540120442 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 218.76171875 disk.rbd1.out‚ 41.994140625) disk./.usagePercent‚70.51348451317465% disk./.usageGB‚1260.8527183532715 disk.drbd2111.in‚0& proc.memory.availableMB‚ 1930.46875 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 80.203125$àZ 2ü- _wandbruntime‚8385  _runtime‚8385 R;¦ÇýÁ & lr-AdamW/pg1‚9.074914106195272e-05 & lr-AdamW/pg2‚9.074914106195272e-05 trainer/global_step‚4599 % _timestamp‚1.7848476226382954e+09 _runtime‚8385.995525657 _step‚380¨Ú'ôÌþÆ trainer/global_step‚4599 % _timestamp‚1.7848476226382954e+09 _runtime‚8385.995525657 _step‚380 & lr-AdamW/pg1‚9.074914106195272e-05 & lr-AdamW/pg2‚9.074914106195272e-05ü ‹c\2ÿ- _wandbruntime‚8386  _runtime‚8386­E?‰(€ýO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12258980 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1328841771393536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1328311057745072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚381 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26255564 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26255201 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4798777942016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1328840399783936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39923520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1324041621841920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1323518970938352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13996584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82624719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1328840399783936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚358 % _timestamp‚1.7848476226393414e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1328309687705818 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1328841771393536 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39923526 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42711263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42711263 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1324041621841920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4800123378176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚368340294982144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚363 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39923526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12258975 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13996226 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42701199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5867824266752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968¥ïŒ[‚P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1328840399783936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39923520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1324041621841920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1323518970938352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.smc‚kæ­#all_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13996584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82624719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1328840399783936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚358 % _timestamp‚1.7848476226393414e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1328309687705818 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1328841771393536 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39923526 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42711263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42711263 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1324041621841920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4800123378176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚368340294982144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚363 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39923526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12258975 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13996226 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42701199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5867824266752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12258980 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1328841771393536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1328311057745072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚381 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26255564 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26255201 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4798777942016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ýçX@ 2‚- _wandbruntime‚8387  _runtime‚83871D0ç 'ƒ‡N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362551425172480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1328589510281234 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42720359 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1324320531132416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362551057081344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1324320531132416 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4801145893376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5869079007744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13999493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4799800450048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5869132364800 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1328590880320600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39931887 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4793107608472 _step‚382 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚4599 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82652252 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82642168 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12261615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26260745 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1323797747284160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848476241170719e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368420136089088_Õ»'„ŒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1324320557305856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42710281 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4801145893376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5869079007744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13999493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4799800450048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5869132364800 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1328590880320600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39931887 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4793107608472 _step‚382 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚4599 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82652252 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82642168 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12261615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26260745 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1323797747284160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848476241170719e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368420136089088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362551425172480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1328589510281234 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42720359 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1324320531132416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362551057081344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1324320531132416 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1342837053864960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5930548365312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372332756736512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚363 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848477094624157e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _runtime‚8472.821570484 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1342838425474560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4841846078242 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4851312934400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1337987112540160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181»ÿ(—‚P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1342838425474560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4841846078242 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4851312934400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1337987112540160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40341870 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83509091 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43157151 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83499021 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1337459059627152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4843190689528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _step‚385 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12390980 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14144673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1337459034199184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40341876 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1337987086366720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40341876 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366401789012992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5930599632384 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1342302250316680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43157151 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1342837053864960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5930548365312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372332756736512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚363 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848477094624157e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _runtime‚8472.821570484 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4850990006272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26541143 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842866879882 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26540778 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40350237 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5931802556416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14147163 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5931855913472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 B8DeviceStats~ŸÔhMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14147523 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1338266021830656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1338265995657216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4649 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1343116985663488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372412599383552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1342582072892208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848477109413612e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12393620 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372412177935360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40350243 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40350237 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43176311 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4852335449600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83526554 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40350237 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43176311 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4852335449600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83526554 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4850990006272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26541143 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842866879882 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26540778 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40350237 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5931802556416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14147163 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5931855913472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14147523 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1338266021830656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1338265995657216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4649 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1343116985663488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372412599383552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1342582072892208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005‚Ãaû 2›- _wandbruntime‚8474  _runtime‚84748ª„ œ„ 1loss/SRqqt1/detection_loss‚0.9451812411914341 /loss/FBt1/detection_loss‚0.6612773323204459 trainer/global_step‚4649 % _timestamp‚1.7848477109755368e+09 _runtime‚8474.332815245 _step‚387 2loss/SRqqt1/assignment_loss‚0.5493038334874214 2loss/SRqqt2/assignment_loss‚0.5588074627215731 /loss/FRt2/detection_loss‚1.4739084311771427 0loss/FBt2/detection_loss‚0.44894169262055156 %loss/total_loss‚8.423275637434113 epoch‚16 /loss/FRt1/detection_loss‚0.5306644689033423 0loss/FRt1/assignment_loss‚1.8045176104091762 0loss/FRt2/assignment_loss‚0.7616386931933714 2loss/FBt1/assignment_loss‚0.011996344292944836 1loss/FBt2/assignment_loss‚0.05889680092617547 1loss/SRqqt2/detection_loss‚0.61814172619053565+^¤‰ 1loss/FBt2/assignment_loss‚0.05889680092617547 1loss/SRqqt2/detection_loss‚0.6181417261905356 1loss/SRqqt1/detection_loss‚0.9451812411914341 /loss/FBt1/detection_loss‚0.6612773323204459 trainer/global_step‚4649 % _timestamp‚1.7848477109755368e+09 _runtime‚8474.332815245 _step‚387 2loss/SRqqt1/assignment_loss‚0.5493038334874214 2loss/SRqqt2/assignment_loss‚0.5588074627215731 /loss/FRt2/detection_loss‚1.4739084311771427 0loss/FBt2/detection_loss‚0.44894169262055156 %loss/total_loss‚8.423275637434113 epoch‚16 /loss/FRt1/detection_loss‚0.5306644689033423 0loss/FRt1/assignment_loss‚1.8045176104091762 0loss/FRt2/assignment_loss‚0.7616386931933714 2loss/FBt1/assignment_loss‚0.011996344292944836ƒÆKlCŸž‚(:” ߺŠÓ»¡á³ gpu.0.gpu‚79.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.377' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.45900000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘‘Ù:Ÿ‚(:¯  ߺŠÓùý³ disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 network.recv‚488908 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 network.sent‚ 8.295138e+06 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚27402.0302734375 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 cpu‚25.315662537412003$ disk./.usageGB‚1260.849594116211 disk.sda.out‚ 220.8515625 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2041.in‚0$ memory_percent‚92.14649200439453 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 220.8515625 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.nvme0n1.in‚ 80.203125 proc.cpu.threads‚17) disk./.usagePercent‚70.51330978948299 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1226.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0' proc.memory.availableMB‚ 1930.078125 disk.dm-0.in‚ 80.203125* proc.memory.percent‚11.829026540120442 disk.sda.in‚0.03125" disk.dm-0.out‚27402.0302734375 disk.drbd2111.in‚0 disk.rbd1.out‚ 42.41015625? ‚(:´  Ó¡Ì€´ disk.sda.out‚ 221.2890625& disk.nvme0n1.out‚27402.32763671875 proc.cpu.threads‚17) proc.memory.availableMB‚ 1929.86328125 disk.drbd1931.in‚0 network.recv‚489895% disk./.usageGB‚1260.8495407104492 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 network.sent‚ 8.300799e+06 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.dm-0.in‚ 80.203125 disk.drbd1828.in‚0$ memory_percent‚92.14736620585123 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2907.1015625* proc.memory.percent‚11.829026540120442 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1817.in‚0# disk.dm-0.out‚27402.32763671875 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51330680275323 disk.drbd1584.out‚0 disk.sda2.out‚ 221.2890625 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 42.47265625 disk.drbd1226.in‚0 disk.drbd1510.out‚0 cpu‚25.315546848650374 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 80.203125ˆë[Nž¡‚(:“ Ó¢„à³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.674' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.89133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.WƒLž¢‚(:“ ýºŠÓ¾ŒÖ³ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚265.681' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.56033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰êTY<£‚(:±  ýºŠÓ‡®ù³$ disk./.usageGB‚1260.849521636963 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 221.5859375 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1938.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.14728673299155 cpu‚25.315490590586514 disk.drbd1289.in‚0 disk.drbd1285.out‚0 network.recv‚491035 disk.sda.in‚0.03125 disk.sda.out‚ 221.5859375 disk.drbd1931.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚27402.50341796875 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 80.203125 disk.rbd1.out‚ 42.47265625) disk./.usagePercent‚70.51330573606403 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚27402.50341796875 disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 80.203125 network.sent‚ 8.36092e+06 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0( proc.memory.availableMB‚ 1929.8828125 disk.sda2.in‚0.03125 disk.sda1.out‚0* proc.memory.percent‚11.829026540120442³Åÿœ¤‚(:‘ Œ»ŠÓ΀ܳ gpu.0.gpu‚5.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.283' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.09433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14292023 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5993203030016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1356833707945984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14291664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1351399148315952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5993155957248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚8559.893916553 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1351932550891520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84373323 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4901157054464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40760226 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1356833707945984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1351932577064960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84383393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚364 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4894294572336 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14292023 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5993203030016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1356833707945984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14291664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1351399148315952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5993155957248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚8559.893916553 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1351932550891520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84373323 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4901157054464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40760226 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1356833707945984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1351932577064960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84383393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚364 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4894294572336 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40768593 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5994398572032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4895315374088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84400856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12525615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43632263 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43622185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40768587 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1352211486355456 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1357115011361280 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1352211460182016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4903525005824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376404092248576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4902179562496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1356571895424450 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40768587 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14294971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43632263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14294609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _step‚390 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40768593 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370409693676544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84400856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26820224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848477980053005e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84390772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1351677950089728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43622185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370410061767680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1351677924661760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4893970762690 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84390772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1351677950089728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43622185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370410061767680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1351677924661760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4893970762690 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40768593 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inac*šCBðtive_split_bytes.small_pool.freed‚ 5994398572032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4895315374088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84400856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12525615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43632263 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43622185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40768587 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1352211486355456 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1357115011361280 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1352211460182016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4903525005824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376404092248576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4902179562496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1356571895424450 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40768587 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14294971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43632263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14294609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _step‚390 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40768593 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370409693676544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84400856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26820224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848477980053005e+09 F(ÆýO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1370830362027008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44069055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14439737 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 _step‚393 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1365878041589760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12654980 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1370284635459896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚374260793699328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1365339211576784 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1370830362027008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4945398455144 % _timestamp‚1.7848478831524317e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6055735661056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1365878015416320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4944053843858 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4953692046848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4953692046848 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12654975 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27094712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380316161269248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1370831733636608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41178570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44079119 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚4749 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1365878015416320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44079119 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85257695 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380316576433152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41178570 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1365878041589760 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1370830362027008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44069055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14439737 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 _step‚393 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1365878041589760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12654980 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1370284635459896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚374260793699328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1365339211576784 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1370830362027008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4945398455144 % _timestamp‚1.7848478831524317e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6055735661056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1365878015416320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4944053843858 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4953692046848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4953692046848 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12654975 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27094712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380316161269248‰Caö¶2È- _wandbruntime‚8647  _runtime‚8647¬,”š 'ɇN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85265074 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4945074645498 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27100325 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1366156950880256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44078137 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374339011974144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4954714562048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14442710 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1366156950880256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14443072 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380396007372288 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8647.984140108 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1371111665442304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4946419256896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380396426723328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85265074 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44088215 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85275158 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41186937 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1370564458035424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12657620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85275158 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44088215 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1366156924706816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41186937 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8647.984140108 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1371111665442304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4946419256896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380396426723328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85265074 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44088215 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85275158 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41186937 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1370564458035424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12657620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85275158 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44088215 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1366156924706816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41186937 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85265074 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4945074645498 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27100325 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1366156950880256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44078137 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374339011974144Š‡{O2Ë- _wandbruntime‚8648  _runtime‚8648º—ÏL ̃ 0loss/FRt1/assignment_loss‚1.8380460378974073 trainer/global_step‚4749 # _timestamp‚1.78484788466661e+09 2loss/SRqqt2/assignment_loss‚0.5662852520689583 3loss/FBt1/assignment_loss‚0.0067144522040580926 2loss/FBt2/assignment_loss‚0.038579383634890954 1loss/SRqqt2/detection_loss‚0.6481064948800223 0loss/FBt1/detection_loss‚0.48907572472648275 /loss/FBt2/detection_loss‚0.4491732999732535 %loss/total_loss‚8.292265015555254 2loss/SRqqt1/assignment_loss‚0.6263676543249344 /loss/FRt1/detection_loss‚0.5550350994400374 1loss/SRqqt1/detection_loss‚0.8894267440831393 epoch‚16 /loss/FRt2/assignment_loss‚0.714900062765394 /loss/FRt2/detection_loss‚1.4705548095566752 _runtime‚8648.023808408 _step‚395ÎІ¦Íˆ epoch‚16 /loss/FRt2/assignment_loss‚0.714900062765394 /loss/FRt2/detection_loss‚1.4705548095566752 _runtime‚8648.023808408 _step‚395 0loss/FRt1/assignment_loss‚1.8380460378974073 trainer/global_step‚4749 # _timestamp‚1.78484788466661e+09 2loss/SRqqt2/assignment_loss‚0.5662852520689583 3loss/FBt1/assignment_loss‚0.0067144522040580926 2loss/FBt2/assignment_loss‚0.038579383634890954 1loss/SRqqt2/detection_loss‚0.6481064948800223 0loss/FBt1/detection_loss‚0.48907572472648275 /loss/FBt2/detection_loss‚0.4491732999732535 %loss/total_loss‚8.292265015555254 2loss/SRqqt1/assignment_loss‚0.6263676543249344 /loss/FRt1/detection_loss‚0.5550350994400374 1loss/SRqqt1/detection_loss‚0.8894267440831393‹gŸ žÎ‚(:“ “¼ŠÓñ«å³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.317' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.10566666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°›‰6@Ï‚(:µ  “¼ŠÓ̾…´ disk.rbd1.out‚ 43.6923828125% disk./.usageGB‚1260.8500747680664 disk.drbd1289.in‚0 disk.drbd1965.out‚0% disk.nvme0n1.out‚27406.0732421875 disk.drbd1510.out‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 1930.1875* proc.memory.percent‚11.829026540120442 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51333667005095 disk.drbd2118.in‚0 disk.sda.out‚ 225.26171875 cpu‚25.31467587910778 disk.drbd1828.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1673.out‚0 network.recv‚496017 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 80.54296875 disk.sda.in‚0.03125 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0$ memory_percent‚92.14604695638022 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.dm-0.in‚ 80.54296875 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 network.sent‚ 8.438298e+06 disk.drbd2111.out‚0 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2907.1015625" disk.dm-0.out‚27406.0732421875 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 225.26171875 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0@ò²·’Ђ(:‡ ¢¼ŠÓر³³ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚330.858' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚110.286 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚055›¶Ñ‚(:¶  ¢¼ŠÓ¾ªÔ³% disk./.usageGB‚1260.8500175476074 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 cpu‚25.315830593396658 disk.drbd2118.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 225.54296875 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.sda.out‚ 225.54296875% proc.memory.availableMB‚ 1930.4375 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51333346998334 disk.drbd1510.out‚0 disk.drbd155Ò[E΋0.in‚0$ memory_percent‚92.14502970377603 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 43.8408203125 network.sent‚ 8.446466e+06 disk.dm-0.in‚ 80.54296875 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0" disk.dm-0.out‚27407.0205078125 disk.drbd1356.out‚0 disk.drbd2041.in‚0 network.recv‚497149 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0% disk.nvme0n1.out‚27407.0205078125 disk.drbd1552.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 80.54296875 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.829026540120442öÖqç2Ò- _wandbruntime‚8684  _runtime‚8684j¸Ð§r:Óìt *jet/accuracy_1_of_4‚0.9758731126785278 )jet/accuracy_2_of_4‚0.820432186126709 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47727271914482117 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17454245686531067 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7581018209457397 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2563517987728119 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1863025426864624 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20113514363765717 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22665907442569733 +jet/accuracy_2_of_2‚0.15501679480075836 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2023787498474121 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7146731019020081 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893012046813965 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6405767202377319 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8695411682128906 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3962508738040924 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7823588252067566 particle/accuracy_1_of_1‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7817099690437317 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905175566673279 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6995307803153992 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7620640397071838 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12518398463726044 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6584141850471497 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03183945640921593 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6294870972633362 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 +jet/accuracy_3_of_4‚0.11704510450363159 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28970280289649963 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7579178810119629 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03202197700738907 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24033977091312408 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7958412170410156 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6681671738624573 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11033041775226593 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6551182866096497 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7965007424354553 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029471704736351967 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9276252388954163 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5925986170768738 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.759943425655365 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6628156900405884 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7064353227615356 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6709111928939819 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8130969405174255 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03202197700738907 ,jet/accuracy_4_of_4‚0.018756305798888206 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5569698214530945 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22665907442569733 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7384615540504456 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7248656153678894 +jet/accuracy_3_of_3‚0.06871245801448822 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21892540156841278 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28873375058174133 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24046416580677032 /particle/accuracy_3_of_4‚0.9964285492897034 epoch‚16 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09019029140472412 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5969314575195312 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927783012390137 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.729354739189148 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.592484176158905 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02090262994170189 +particle/specificity‚0.9086844325065613 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.688663899898529 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2407902628183365 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 6validation_average_jet_accuracy‚0.3750653564929962 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.76036536693573 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2299196869134903 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22540511190891266 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8570465445518494 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7105639576911926 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7614334225654602 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4287322163581848 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5307692289352417 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10459425300359726 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7940707802772522 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4211747348308563 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964697957038879 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2262391895055771 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8570465445518494 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7669504880905151 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763131320476532 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 .particle/accuracy_4_of_4‚0.968994677066803 % _timestamp‚1.7848479216345787e+09 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03090975619852543 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5844318270683289 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03950032219290733 _step‚396 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6653546690940857 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07739640772342682 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_1_of_3‚0.9317444562911987 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18588507175445557 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6809015870094299 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3966819643974304 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038920968770980835 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18779566884040833 (particle/accuracy‚0.8511614203453064 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04218580946326256 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7674034833908081 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974713921546936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46680164337158203 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24208377301692963 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7238238453865051 )loss/val_total_loss‚8.534171981077048 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02932204306125641 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6487306952476501 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6393888592720032 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 _runtime‚8684.992661821 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9119271636009216 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18929459154605865 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7633755803108215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.814258873462677 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 trainer/global_step‚4759 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7941557765007019 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2119123488664627 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046576328575611115 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6629908680915833 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6476263999938965 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.325945109128952 6"Purity/0FRt2SRqqt0FBt/event_purity‚0.4372578561306 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.30018314719200134 *jet/accuracy_2_of_3‚0.5632179975509644 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6942652463912964 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22184345126152039 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3342226445674896 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1875445544719696 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870508909225464 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6514164805412292 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5004706382751465 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4572626054286957 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1875610053539276 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7520008087158203 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.858410656452179 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6709111928939819 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7102798819541931 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625ÛºÊw:Ôñt ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07739640772342682 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_1_of_3‚0.9317444562911987 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18588507175445557 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6809015870094299 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3966819643974304 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038920968770980835 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18779566884040833 (particle/accuracy‚0.8511614203453064 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04218580946326256 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7674034833908081 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974713921546936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46680164337158203 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24208377301692963 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7238238453865051 )loss/val_total_loss‚8.534171981077048 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02932204306125641 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6487306952476501 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6393888592720032 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 _runtime‚8684.992661821 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9119271636009216 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18929459154605865 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7633755803108215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.814258873462677 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 trainer/global_step‚4759 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7941557765007019 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2119123488664627 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046576328575611115 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6629908680915833 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6476263999938965 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.325945109128952 6"Purity/0FRt2SRqqt0FBt/event_purity‚0.4372578561306 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.30018314719200134 *jet/accuracy_2_of_3‚0.5632179975509644 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6942652463912964 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22184345126152039 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3342226445674896 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1875445544719696 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870508909225464 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6514164805412292 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5004706382751465 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4572626054286957 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1875610053539276 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7520008087158203 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.858410656452179 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6709111928939819 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7102798819541931 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 *jet/accuracy_1_of_4‚0.9758731126785278 )jet/accuracy_2_of_4‚0.820432186126709 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47727271914482117 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17454245686531067 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7581018209457397 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2563517987728119 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1863025426864624 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20113514363765717 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22665907442569733 +jet/accuracy_2_of_2‚0.15501679480075836 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2023787498474121 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7146731019020081 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893012046813965 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6405767202377319 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8695411682128906 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3962508738040924 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7823588252067566 particle/accuracy_1_of_1‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7817099690437317 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905175566673279 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6995307803153992 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7620640397071838 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12518398463726044 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6584141850471497 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03183945640921593 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6294870972633362 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 +jet/accuracy_3_of_4‚0.11704510450363159 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28970280289649963 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7579178810119629 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03202197700738907 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24033977091312408 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7958412170410156 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6681671738624573 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11033041775226593 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6551182866096497 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7965007424354553 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029471704736351967 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9276252388954163 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5925986170768738 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.759943425655365 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6628156900405884 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7064353227615356 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6709111928939819 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8130969405174255 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03202197700738907 ,jet/accuracy_4_of_4‚0.018756305798888206 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5569698214530945 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22665907442569733 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7384615540504456 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7248656153678894 +jet/accuracy_3_of_3‚0.06871245801448822 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21892540156841278 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28873375058174133 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24046416580677032 /particle/accuracy_3_of_4‚0.9964285492897034 epoch‚16 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09019029140472412 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5969314575195312 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927783012390137 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.729354739189148 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.592484176158905 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02090262994170189 +particle/specificity‚0.9086844325065613 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.688663899898529 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2407902628183365 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 6validation_average_jet_accuracy‚0.3750653564929962 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.76036536693573 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2299196869134903 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22540511190891266 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8570465445518494 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7105639576911926 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7614334225654602 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4287322163581848 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5307692289352417 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10459425300359726 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7940707802772522 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4211747348308563 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964697957038879 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2262391895055771 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8570465445518494 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7669504880905151 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763131320476532 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 .particle/accuracy_4_of_4‚0.968994677066803 % _timestamp‚1.7848479216345787e+09 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03090975619852543 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5844318270683289 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03950032219290733 _step‚396 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6653546690940857 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1Œ$i|žÕ‚(:“ ±¼ŠÓóùγ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚106.073' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.35766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢lg’Ö‚(:§  ±¼ŠÓ¿Ùï³ disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.drbd2041.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 1929.91796875 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1673.out‚0 disk.dm-0.in‚81.5 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 cpu‚25.32010319122469 disk.drbd1356.in‚0" disk.dm-0.out‚27407.2041015625 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 226.078125) disk./.usagePercent‚70.51333346998334 disk.drbd1965.out‚0 disk.rbd1.in‚0$ memory_percent‚92.14714368184408 disk.sda1.out‚0 disk.sda2.out‚ 226.078125 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.829026540120442" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 8.446532e+06 disk.drbd1817.in‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8500175476074 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 network.recv‚497215 disk.drbkp„ d1828.out‚0 disk.rbd1.out‚ 43.8408203125 disk.drbd1285.in‚0% disk.nvme0n1.out‚27407.2041015625 disk.sda.in‚0.03125 disk.nvme0n1.in‚81.5Å¿]Ϟׂ(:“ À¼ŠÓÞùå³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚256.568' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.52266666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ÿ¦G5Ø‚(:ª  À¼ŠÓô„´ disk.sda1.out‚0 disk.drbd1510.in‚0 disk.sda.out‚ 226.48828125 disk.drbd2041.in‚0 disk.drbd1285.in‚0$ memory_percent‚88.14571698506674 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 2913.30859375 disk.sda.in‚0.03125 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1584.in‚0 disk.dm-0.in‚81.5" disk.dm-0.out‚28377.4931640625 disk.rbd1.out‚ 43.8408203125 disk.nvme0n1.in‚81.5 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 network.recv‚497323 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2111.out‚0" proc.memory.rssMB‚ 2907.1015625$ disk./.usageGB‚1260.853816986084 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51354595447282 disk.sda2.out‚ 226.48828125 disk.drbd1289.in‚0% disk.nvme0n1.out‚28377.4931640625 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 cpu‚25.318579188836534 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 network.sent‚ 8.44664e+06H¸ÇžÙ‚(:“ ϼŠÓÓ‰Ù³ gpu.0.gpu‚76.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚59.0 gpu.0.powerWatts‚107.042' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.68066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SYfG2Ú‚(:§  ϼŠÓì·ö³ disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚28377.6728515625 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0$ memory_percent‚88.14671834309897 disk.drbd1510.out‚0% proc.memory.availableMB‚ 2913.0625 disk.drbd1828.out‚0* proc.memory.percent‚11.829026540120442 disk.dm-0.in‚81.5 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 226.83203125 proc.cpu.threads‚17" disk.dm-0.out‚28377.6728515625) disk./.usagePercent‚70.51354723449987 disk.drbd1931.out‚0" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 8.509311e+06% disk./.usageGB‚1260.8538398742676 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚81.5 cpu‚25.318462252065544 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 226.83203125 disk.rbd1.out‚ 43.974609375 network.recv‚499216 disk.drbd1356.in‚0—Û‚(:“ Þ¼ŠÓ êª³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚272.739' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.91299999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ.Ü‚(:£  Þ¼ŠÓ—£Ï³ disk.drbd1938.out‚0$ disk./.usageGB‚1260.850040435791 disk.drbd1584.out‚0 disk.sda.out‚ 227.1953125 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 227.1953125% proc.memory.availableMB‚ 2913.0625 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.829026540120442 network.sent‚ 8.509419e+06 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0% disk.nvme0n1.out‚29347.6220703125 disk.drbd1289.out‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚81.5) disk./.usagePercent‚70.51333475001039 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 43.974609375 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.rbd1.in‚0 cpu‚25.31837545571861 disk.drbd1552.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 network.recv‚499324 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125" disk.dm-0.out‚29347.6220703125 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0$ memory_percent‚88.14671834309897 disk.drbd1817.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚81.5 disk.drbd1673.in‚0 disk.drbd1226.out‚0]Âޞ݂(:“ í¼ŠÓð«Ò³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚199.321' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.44033333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w¸µ.Þ‚(:£  í¼ŠÓºñ³ disk.rbd1.in‚0 disk.sda2.out‚ 227.515625 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.dm-0.in‚81.5 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚81.5% disk.nvme0n1.out‚29347.8251953125$ memory_percent‚88.14671834309897% proc.memory.availableMB‚ 2913.0625 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1584.out‚0 disk.sda.out‚ 227.515625 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 cpu‚25.318313972576462" disk.dm-0.out‚29347.8251953125 disk.drbd1673.out‚0 disk.drbd1510.in‚0% disk./.usageGB‚1260.8500022888184 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 44.037109375 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0 network.recv‚499390 disk.drbd2078.out‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.829026540120442) disk./.usagePercent‚70.51333261663196 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 8.509485e+06 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1828.in‚0]­(H2ß- _wandbruntime‚8758  _runtime‚8758hu¹ZÇàÁ % _timestamp‚1.7848479948829381e+09 _runtime‚8758.240180756 _step‚397 & lr-AdamW/pg1‚8.847548178954597e-05 & lr-AdamW/pg2‚8.847548178954597e-05 trainer/global_step‚4799À~RXÌáÆ & lr-AdamW/pg2‚8.847548178954597e-05 trainer/global_step‚4799 % _timestamp‚1.7848479948829381e+09 _runtime‚8758.240180756 _step‚397 & lr-AdamW/pg1‚8.847548178954597e-05;ÕD2â- _wandbruntime‚8758  _runtime‚8758Ï»¸(ãýO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1387393311271936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86258671 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5009154606080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1382384156665856 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1386839067982732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1381838301416752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384365227400192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41686352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚332 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1382384156665856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41686352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚337 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1387393311271936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86268741 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14616168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5002111177266 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44572319 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12793867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6126115890176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41686358 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1382384182839296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14616500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5000766565980 _runtime‚8758.242313102ÛêS(ä‚P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86268741 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14616168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5002111177266 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44572319 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12793867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6126115890176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41686358 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1382384182839296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14616500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5000766565980 _runtime‚8758.242313102 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1387393311271936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86258671 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5009154606080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1382384156665856 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1386839067982732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1381838301416752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384365227400192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41686352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚332 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1382384156665856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41686352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚337 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1387393311271936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Ž2ñßL2å- _wandbruntime‚8759  _runtime‚8759¿»Ez æ†N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1382663065956352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6127364792320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14619449 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011522557440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41694725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1387118890558148 % _timestamp‚1.7848479965340035e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41694719 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010177114112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384445062668288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86286204 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86286204 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44591479 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_byž6z ’tes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12796507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384444643317248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44581401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378317697875968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86276120 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1382663092129792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44581401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚ 8759.89157138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41694719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12796512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5001787367620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1387674614687232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1382663092129792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010177114112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚335 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27415626 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1382117128618496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1382117103190528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44591479 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440MXSI'ç‹N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚335 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27415626 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1382117128618496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1382117103190528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44591479 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1382663065956352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6127364792320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14619449 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011522557440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41694725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1387118890558148 % _timestamp‚1.7848479965340035e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41694719 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010177114112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384445062668288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86286204 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86286204 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44591479 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12796507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384444643317248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44581401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378317697875968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86276120 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1382663092129792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44581401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚ 8759.89157138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41694719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12796512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5001787367620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1387674614687232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1382663092129792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010177114112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0°“22è- _wandbruntime‚8759  _runtime‚8759»ú>O éƒ 0loss/SRqqt1/detection_loss‚0.932689620881879 0loss/FRt2/assignment_loss‚0.8047607078730503 1loss/SRqqt2/detection_loss‚0.6003385815815869 %loss/total_loss‚8.891368700867474 epoch‚17 trainer/global_step‚4799 % _timestamp‚1.7848479965717301e+09 0loss/FRt1/assignment_loss‚1.7665406734750586 1loss/SRqqt1/assignment_loss‚0.641301816001391 2loss/SRqqt2/assignment_loss‚0.5415490479848746 2loss/FBt2/assignment_loss‚0.027915846821704084 /loss/FRt1/detection_loss‚0.5603365886151211 /loss/FBt1/detection_loss‚1.0036625726907342 _runtime‚8759.928925347 _step‚400 /loss/FRt2/detection_loss‚1.4375512689965664 /loss/FBt2/detection_loss‚0.5741471911824192 3loss/FBt1/assignment_loss‚0.0005747847630888503”v)êˆ 3loss/FBt1/assignment_loss‚0.0005747847630888503 0loss/SRqqt1/detection_loss‚0.932689620881879 0loss/FRt2/assignment_loss‚0.8047607078730503 1loss/SRqqt2/detection_loss‚0.6003385815815869 %loss/total_loss‚8.891368700867474 epoch‚17 trainer/global_step‚4799 % _timestamp‚1.7848479965717301e+09 0loss/FRt1/assignment_loss‚1.7665406734750586 1loss/SRqqt1/assignment_loss‚0.641301816001391 2loss/SRqqt2/assignment_loss‚0.5415490479848746 2loss/FBt2/assignment_loss‚0.027915846821704084 /loss/FRt1/detection_loss‚0.5603365886151211 /loss/FBt1/detection_loss‚1.0036625726907342 _runtime‚8759.928925347 _step‚400 /loss/FRt2/detection_loss‚1.4375512689965664 /loss/FBt2/detection_loss‚0.5741471911824192³¥Ž5ë‚(:ª  ü¼ŠÓÿÂÚ³ disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.nvme0n1.in‚81.5 disk.sda.in‚0.03125 disk.drbd2111.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.829026540120442% disk./.usageGB‚1260.8500022888184 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.drbd1584.in‚0$ memory_percent‚88.14751307169595 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚29348.21337890625 disk.drbd1510.in‚0 disk.drbd1673.in‚0# disk.dm-0.out‚29348.21337890625 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 network.sent‚ 8.612279e+06) disk./.usagePercent‚70.51333261663196 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 network.recv‚503314 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1285.in‚0 cpu‚25.318170104802547 disk.dm-0.in‚81.5 disk.rbd1.out‚ 44.037109375 disk.drbd1552.in‚0( proc.memory.availableMB‚ 2912.8671875 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 227.7890625 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 227.7890625 disk.drbd1226.in‚0fÖ±}žì‚(:“ ü¼ŠÓÒ§µ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚272.368' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.78933333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸-ß“í‚(:ˆ ‹½ŠÓòÓ׳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚280.488' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j¸‡8î‚(:­  ‹½ŠÓÜöø³ disk.dm-0.in‚81.5 disk.drbd2078.in‚0 disk.drbd1510.out‚0$ memory_percent‚88.14765612284342 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51336013721344% disk./.usageGB‚1260.8504943847656 disk.drbd1552.in‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚29348.79150390625 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 228.08203125 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.829026540120442 network.sent‚ 8.612345e+06 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0& disk.nvme0n1.out‚29348.79150390625 disk.drbd1550.in‚0 network.recv‚503380 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 2912.83203125 disk.drbd1584.in‚0 cpu‚25.31810934224741 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚81.5" proc.memory.rssMB‚ 2907.1015625 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 228.08203125 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 44.1708984375 proc.cpu.threads‚17 disk.drbd1289.in‚0ÃÎxŸï‚(:” š½ŠÓ½›ç³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.708' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.23600000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒxHh.ð‚(:£  š½ŠÓ·Ê‹´ disk.drbd1356.out‚0 disk.sda2.out‚228.5625 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 44.1708984375 disk.nvme0n1.in‚81.5 disk.drbd1550.in‚0 cpu‚25.318077691251013 network.recv‚505776 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0& disk.nvme0n1.out‚29349.04443359375 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd2041.in‚0( proc.memory.availableMB‚ 1940.6796875 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 network.sent‚ 8.67495e+06) disk./.usagePercent‚70.51336120390265 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.sda.out‚228.5625# disk.dm-0.out‚29349.04443359375 disk.drbd2111.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1356.in‚0$ disk./.usageGB‚1260.850513458252 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.829026540120442 disk.dm-0.in‚81.5 disk.drbd1552.in‚0$ memory_percent‚92.10335413614908ýR•©žñ‚(:“ ©½ŠÓØÛس gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚269.775' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.92499999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦PQ2ò‚(:§  ©½ŠÓœ“ú³) disk./.usagePercent‚70.51335907052423 disk.drbd1931.out‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚81.5 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 network.recv‚506743 disk.drbd1550.out‚0% disk./.usageGB‚1260.8504753112793 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 network.sent‚ 8.675929e+06 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 cpu‚25.317934723026294$ memory_percent‚92.10437138875326* proc.memory.percent‚11.829026540120442# disk.dm-0.out‚29349.70458984375 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚81.5 disk.drbd1817.in‚0 disk.sda.out‚ 228.90625 disk.drbd1289.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1940.4296875 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 228.90625 disk.drbd2041.out‚0 disk.rbd1.out‚ 44.2333984375 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0& disk.nvme0n1.out‚29349.70458984375D5õÏ4ó‚(:©  ¸½ŠÓýèú³ network.recv‚507732 disk.drbd1356.in‚0 disk.rbd1.out‚ 44.2333984375 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚29350.0068359375 disk.drbd1584.in‚0 disk.sda.out‚ 229.16796875 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1965.out‚0 disk.drbd1356.out‚0$ memory_percent‚92.10489590962729" proc.memory.rssMB‚ 2907.1015625 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0" disk.dm-0.out‚29350.0068359375 disk.sda1.out‚0 network.sent‚ 8.678895e+06 disk.drbd2118.in‚0 disk.nvme0n1.in‚81.5 cpu‚25.317876045472616' disk./.usagePercent‚70.513362057254 disk.drbd2118.out‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 1940.30078125 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0$ disk./.usageGB‚1260.850528717041 disk.sda2.out‚ 229.16796875 disk.dm-0.in‚81.5 disk.drbd1550.out‚0 disk.drbd1552.out‚0G°ž'žô‚(:“ ¸½ŠÓ„õس gpu.0.gpu‚96.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚266.919' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.97299999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M<þj0õ‚(:¥  ǽŠÓž²×³ disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚81.5 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0) disk./.usagePercent‚70.51336312394321 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.sda1.out‚0 disk.sda.out‚ 229.46484375 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 cpu‚25.3177628804358 network.sent‚ 8.681156e+06 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0' proc.memory.availableMB‚ 1940.265625 disk.drbd1828.in‚0$ memory_percent‚92.10503896077475" disk.dm-0.out‚29350.3037109375 network.recv‚507930% disk.nvme0n1.out‚29350.3037109375 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 229.46484375 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚81.5% disk./.usageGB‚1260.8505477905273 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1584.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 44.3671875§X·¿œö‚(:‘ ǽŠÓÓ¢µ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.202' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.06733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jé&š2÷- _wandbruntime‚8845  _runtime‚8845Õ‚atÇøÁ & lr-AdamW/pg1‚8.789707776732302e-05 & lr-AdamW/pg2‚8.789707776732302e-05 trainer/global_step‚4849 % _timestamp‚1.7848480819645038e+09 _runtime‚8845.321800964 _step‚401 ‚î—ÌùÆ & lr-AdamW/pg1‚8.789707776732302e-05 & lr-AdamW/pg2‚8.789707776732302e-05 trainer/global_step‚4849 % _timestamp‚1.7848480819645038e+09 _runtime‚8845.321800964 _step‚401‘Èjä2ú- _wandbruntime‚8845  _runtime‚8845Ž*b9ªûýO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14764021 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382168429807616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_staryY t/active_bytes.small_pool.freed‚ 5060344162304 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1396329621190656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1401391336962560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87132973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 _step‚402 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5053215060074 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42104708 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45038335 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27689893 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14763660 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6188654335488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5051870448788 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1395778390105552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12925867 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5061689598464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42104708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6188707699712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1401389965352960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1395778415533520 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1400830260554340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45028271 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388357137507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87143043 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1401389965352960 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42104702 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5061689598464 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14764021 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382168429807616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5060344162304 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1396329621190656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1401391336962560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87132973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 _step‚402 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5053215060074 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42104708 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45038335 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27689893 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14763660 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6188654335488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5051870448788 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1395778390105552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12925867 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5061689598464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42104708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6188707699712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1401389965352960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1395778415533520 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1400830260554340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45028271 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388357137507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87143043 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1401389965352960 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42104702 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5061689598464 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8846.808652662 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27695063 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6189953470464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5062712113664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42113069 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388436546098688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1396608556654592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1396057191879328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45037353 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87160506 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87150422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1401110083129756 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1401669897151488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42113075 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5061366670336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14766912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388436969644032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87160506 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12928512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12928507 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5061366670336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14766556 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5054235861826 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366Õ"e=ÿŒN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14766556 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5054235861826 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.larþÿ¦Õge_pool.allocated‚1396057217307296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42113069 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5052891250428 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45047431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8846.808652662 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27695063 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6189953470464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5062712113664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42113069 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388436546098688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1396608556654592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1396057191879328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45037353 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87160506 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87150422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1401110083129756 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1401669897151488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42113075 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5061366670336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14766912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388436969644032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87160506 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12928512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12928507 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5061366670336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005“ìÁ€ö2€- _wandbruntime‚8846  _runtime‚8846œ4Ø epoch‚17 1loss/SRqqt1/detection_loss‚1.0364603474117624 0loss/FBt2/detection_loss‚0.40891137744200534 %loss/total_loss‚8.595054583420339 _runtime‚8846.839094458 0loss/FRt2/assignment_loss‚0.7807825979530943 /loss/FBt1/detection_loss‚0.7719766824092302 0loss/FRt1/assignment_loss‚1.7240940959129518 2loss/SRqqt2/assignment_loss‚0.5306112560212134 1loss/FBt2/assignment_loss‚0.02032958819096648 /loss/FRt2/detection_loss‚1.5400925819150806 1loss/SRqqt2/detection_loss‚0.6164988377487503 % _timestamp‚1.7848480834818504e+09 _step‚404 2loss/SRqqt1/assignment_loss‚0.6177929900698295 0loss/FBt1/assignment_loss‚0.0162155121422393 .loss/FRt1/detection_loss‚0.531288716203216 trainer/global_step‚4849‘¥™´ ‚† epoch‚17 1loss/SRqqt1/detection_loss‚1.0364603474117624 0loss/FBt2/detection_loss‚0.40891137744200534 %loss/total_loss‚8.595054583420339 _runtime‚8846.839094458 0loss/FRt2/assignment_loss‚0.7807825979530943 /loss/FBt1/detection_loss‚0.7719766824092302 0loss/FRt1/assignment_loss‚1.7240940959129518 2loss/SRqqt2/assignment_loss‚0.5306112560212134 1loss/FBt2/assignment_loss‚0.02032958819096648 /loss/FRt2/detection_loss‚1.5400925819150806 1loss/SRqqt2/detection_loss‚0.6164988377487503 % _timestamp‚1.7848480834818504e+09 _step‚404 2loss/SRqqt1/assignment_loss‚0.6177929900698295 0loss/FBt1/assignment_loss‚0.0162155121422393 .loss/FRt1/detection_loss‚0.531288716203216 trainer/global_step‚4849”×`>žƒ‚(:“ Ö½ŠÓøÇÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.69' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.22999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À¢G?6„‚(:«  Ö½ŠÓØ÷ø³ disk.drbd1828.out‚0 disk.rbd1.out‚ 44.3671875$ memory_percent‚92.10530916849771) proc.memory.availableMB‚ 1940.19921875 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0# disk.dm-0.out‚29350.65576171875 disk.drbd1226.in‚0 cpu‚25.317703171874747 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 229.76171875 disk.sda2.out‚ 229.76171875 disk.dm-0.in‚81.5 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚29350.65576171875 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.nvme0n1.in‚81.5 network.recv‚508038 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0* proc.memory.percent‚11.829026540120442 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0% disk./.usageGB‚1260.8505668640137 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 network.sent‚ 8.681264e+06) disk./.usagePercent‚70.51336419063242Hlßž…‚(:“ 彊ÓÒÓȳ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.627' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.87566666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å<ó 6†‚(:«  彊ÓÝç³ disk.sda.out‚ 230.09765625) disk./.usagePercent‚70.51336312394321 disk.dm-0.in‚81.5 disk.drbd1285.in‚0 disk.drbd2118.out‚0# disk.dm-0.out‚29350.89013671875 disk.drbd1828.in‚0$ memory_percent‚92.10737546284994* proc.memory.percent‚11.829026540120442 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.8505477905273 disk.nvme0n1.in‚81.5 disk.drbd1673.out‚0 disk.sda2.in‚0.03125" proc.memory.rssMB‚ 2907.1015625 disk.rbd1.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 network.recv‚508947 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 230.09765625 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1939.69140625 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0& disk.nvme0n1.out‚29350.89013671875 disk.drbd1584.out‚0 cpu‚25.317617935836175 network.sent‚ 8.681706e+06 disk.rbd1.out‚ 44.4296875 disk.drbd1931.out‚0Ƈȭ'‡‚(:œ  ô½ŠÓ£µê³ disk.sda1.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0' proc.memory.availableMB‚ 1940.421875" proc.memory.rssMB‚ 2907.1015625 cpu‚25.317504039386854 proc.cpu.threads‚17 disk.drbd1828.in‚0 network.sent‚ 8.681772e+06 disk.drbd1226.out‚0 disk.dm-0.in‚81.5 disk.drbd2041.in‚0 disk.nvme0n1.in‚81.5 disk.drbd1965.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.10440317789713 network.recv‚509013 disk.rbd1.out‚ 44.5234375* proc.memory.percent‚11.829026540120442 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚29351.24609375 disk.drbd2118.out‚0 disk.sda2.out‚ 230.40625 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0$ disk./.usageGB‚1260.850570678711 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.out‚ 230.40625 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.dm-0.out‚29351.24609375 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51336440397026cô3ôŸˆ‚(:” ô½ŠÓ»ˆÑ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚153.553' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚51.184333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=j¶§*‰‚(:Ÿ  ƒ¾ŠÓ‘Œƒ´ disk.dm-0.in‚81.5# disk.nvme0n1.out‚29351.41796875 disk.drbd1550.in‚0# memory_percent‚92.1030362447103 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51339277790308 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 cpu‚25.317473492483476 disk.drbd2118.out‚0 disk.sda.out‚ 230.72265625 disk.drbd2041.out‚0 disk.rbd1.out‚44.6875 disk.drbd1584.out‚0* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0( proc.memory.availableMB‚ 1940.7578125 network.recv‚509121 network.sent‚ 8.68188e+06 disk.drbd1673.out‚0 disk.dm-0.out‚29351.41796875 disk.rbd1.in‚0 disk.nvme0n1.in‚81.5 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8510780334473 disk.sda2.out‚ 230.72265625 disk.drbd2078.out‚0 disk.drbd1226.out‚0û&ôIžŠ‚(:“ ƒ¾ŠÓŒÙå³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚267.142' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.04733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“CƒLž‹‚(:“ ’¾ŠÓ£ Ý³ gpu.0.gpu‚40.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.549' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.84966666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nâ[31Œ‚(:¦  ’¾ŠÓÉÔû³ disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51339277790308 disk.sda2.in‚0.03125 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚81.5% disk.nvme0n1.out‚29351.7998046875* proc.memory.percent‚11.829026540120442 disk.drbd1673.in‚0 disk.drbd1938.in‚0" disk.dm-0.out‚29351.7998046875 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 network.sent‚ 8.681988e+06 disk.sda2.out‚ 231.12890625 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2907.1015625 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd2111.out‚0 cpu‚25.31747250963219 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.dm-0.in‚81.5 disk.drbd2041.in‚0 disk.drbd1550.out‚0 network.recv‚509229 disk.sda.out‚ 231.12890625 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 44.8671875# memory_percent‚92.1030362447103% disk./.usageGB‚1260.8510780334473 disk.drbd1817.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1940.7578125ºkPùž‚(:“ ¡¾ŠÓŽï¼³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.946' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.31533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g†û/Ž‚(:¤  ¡¾ŠÓîÚâ³ disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1817.out‚0 network.sent‚ 8.757486e+06 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 231.546875 disk.drbd1931.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚81.5 disk.drbd1828.in‚0 disk.nvme0n1.in‚81.5 disk.drbd1510.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.10305213928223 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8510398864746 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 44.9296875 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 network.recv‚511855* proc.memory.percent‚11.829026540120442 disk.drbd2111.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1940.75390625 disk.drbd1817.in‚0 disk.sda.in‚0.03125% disk.nvme0n1.out‚29351.9912109375 disk.sda2.out‚ 231.546875 cpu‚25.31736064856228 disk.drbd2118.in‚0) disk./.usagePercent‚70.51339064452468 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0" disk.dm-0.out‚29351.9912109375 disk.drbd1965.out‚0!\Á42- _wandbruntime‚8932  _runtime‚8932PÇ¡oÇÁ & lr-AdamW/pg1‚8.731482891770741e-05 & lr-AdamW/pg2‚8.731482891770741e-05 trainer/global_step‚4899 % _timestamp‚1.7848481689925768e+09 _runtime‚8932.349827029 _step‚4056”áÌ‘Æ % _timestamp‚1.7848481689925768e+09 _runtime‚8932.349827029 _step‚405 & lr-AdamW/pg1‚8.731482891770741e-05 & lr-AdamW/pg2‚8.731482891770741e-05 trainer/global_step‚4899•Þt2Ò2’- _wandbruntime‚8932  _runtime‚893285(“üO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1415386619433984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42523052 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5111533718528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6251440904192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1415387991043584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13057872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5111533718528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45494287 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27968614 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1409718504222320 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27968983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5104318942882 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386097748105216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392349189009408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386097380014080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88007275 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45494287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5102974331596 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚406 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1410275085715456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784848168993671e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1415387991043584 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13057867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5112879154688 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1410275111888896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6251381248512 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1414822823165202 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1415386619433984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1410275085715456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392348761262592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8932.354922613 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42523058{‹„T ”P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1415387991043584 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13057867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5112879154688 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1410275111888896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6251381248512 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1414822823165202 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1415386619433984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1410275085715456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392348761262592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8932.354922613 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42523058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1415386619433984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42523052 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5111533718528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6251440904192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1415387991043584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13057872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5111533718528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45494287 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27968614 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1409718504222320 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27968983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5104318942882 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386097748105216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392349189009408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386097380014080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88007275 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45494287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5102974331596 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚406 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1410275085715456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784848168993671e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064–õ¾8Ú2•- _wandbruntime‚8933  _runtime‚8933zz€b '–‡N XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5103995133236 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6252639515136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1410554021179392 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1410553995005952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27974536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88034808 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1410554021179392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45493305 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13060512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42531419 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5113901669888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1410553995005952 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1415101275701364 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45503383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42531425 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27974177 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5112556226560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42531425 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14914024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14913670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88034808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392428605895168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755~ã\ß'—ŒN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42531425 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27974177 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5112556226560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42531425 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14914024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14913670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88034808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392428605895168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5103995133236 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6252639515136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1410554021179392 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1410553995005952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27974536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88034808 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1410554021179392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45493305 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13060512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42531419 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5113901669888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1410553995005952 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1415101275701364 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45503383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968—wJt¤2˜- _wandbruntime‚8933  _runtime‚8933‚Hß™ÿ /loss/FBt2/detection_loss‚0.9016069511201782 epoch‚17 0loss/FRt2/assignment_loss‚0.8156057984932609 2loss/SRqqt1/assignment_loss‚0.5710206576420828 /loss/FRt1/detection_loss‚0.5539174593186396 /loss/FRt2/detection_loss‚1.4934553492490363 %loss/total_loss‚9.209974687548973 trainer/global_step‚4899 $ _timestamp‚1.784848170504576e+09 /loss/FRt1/assignment_loss‚1.818237881363875 2loss/FBt1/assignment_loss‚0.052327537590052486 1loss/FBt2/assignment_loss‚0.02527013524752927 0loss/SRqqt1/detection_loss‚0.972028952647458 1loss/SRqqt2/detection_loss‚0.6305212288774743 /loss/FBt1/detection_loss‚0.8398263001441956 _runtime‚ 8933.86176994 _step‚408 2loss/SRqqt2/assignment_loss‚0.5361564358551905ñ†.Ñ š„ 0loss/FRt2/assignment_loss‚0.8156057984932609 2loss/SRqqt1/assignment_loss‚0.5710206576420828 /loss/FRt1/detection_loss‚0.5539174593186396 /loss/FRt2/detection_loss‚1.4934553492490363 %loss/total_loss‚9.209974687548973 trainer/global_step‚4899 $ _timestamp‚1.784848170504576e+09 /loss/FRt1/assignment_loss‚1.818237881363875 2loss/FBt1/assignment_loss‚0.052327537590052486 1loss/FBt2/assignment_loss‚0.02527013524752927 0loss/SRqqt1/detection_loss‚0.972028952647458 1loss/SRqqt2/detection_loss‚0.6305212288774743 /loss/FBt1/detection_loss‚0.8398263001441956 _runtime‚ 8933.86176994 _step‚408 2loss/SRqqt2/assignment_loss‚0.5361564358551905 /loss/FBt2/detection_loss‚0.9016069511201782 epoch‚17˜wìEÌ-›‚(:¢  °¾ŠÓµ®â³% disk./.usageGB‚1260.8510398864746 disk.drbd1817.in‚0 disk.nvme0n1.in‚81.5 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 45.2890625 disk.drbd1226.out‚0 disk.dm-0.out‚29352.32421875 disk.drbd2118.out‚0 disk.dm-0.in‚81.5 disk.drbd1356.in‚0 disk.drbd1285.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.829026540120442 disk.sda1.in‚0 disk.sda2.in‚0.03125 network.recv‚511921 disk.drbd2041.out‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51339064452468 disk.drbd2078.in‚0$ memory_percent‚92.10319519042969 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 network.sent‚ 8.757552e+06 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0# disk.nvme0n1.out‚29352.32421875 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 1940.71875 cpu‚25.317275935767594 disk.sda2.out‚ 232.12890625 proc.cpu.threads‚17 disk.sda.out‚ 232.12890625 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1550.in‚0”ß žœ‚(:“ °¾ŠÓÆÖó gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚245.213' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.73766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03|¬‚ž‚(:“ ¿¾ŠÓœº¬³ gpu.0.gpu‚53.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚110.51' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.836666666666666 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ß¡¤2ž‚(:£  ¿¾ŠÓÎͳ disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0$ memory_percent‚92.10319519042969 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 network.recv‚512029 disk.drbd1552.in‚0% disk./.usageGB‚1260.8510208129883 disk.drbd1289.in‚0 disk.drbd2078.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51338957783547 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 232.37109375 disk.drbd2111.out‚0 cpu‚25.317275875384823 disk.rbd1.in‚0 disk.drbd1586¨Šü4.in‚0 disk.dm-0.in‚81.5 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0& proc.memory.availableMB‚ 1940.71875 disk.drbd1965.in‚0 disk.drbd1285.out‚0$ disk.nvme0n1.out‚29353.376953125 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 232.37109375 disk.drbd1938.out‚0! disk.dm-0.out‚29353.376953125 network.sent‚ 8.75766e+06 disk.drbd1550.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1828.in‚0 disk.rbd1.out‚ 45.4453125 disk.drbd2111.in‚0 disk.nvme0n1.in‚81.5¶ }žŸ‚(:“ ξŠÓ•ûó gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚267.161' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.05366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þd4. ‚(:£  ξŠÓääݳ disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚81.5 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 cpu‚25.317163047770546 disk.sda.out‚ 232.7578125 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0% proc.memory.availableMB‚ 1940.6875 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 network.sent‚ 8.757726e+06) disk./.usagePercent‚70.51339064452468 disk.drbd1584.out‚0 disk.drbd2041.out‚0% disk./.usageGB‚1260.8510398864746 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚81.5 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 232.7578125 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚45.53125& disk.nvme0n1.out‚29354.28076171875 disk.drbd1226.out‚0 disk.drbd1510.out‚0 network.recv‚512095 disk.sda2.in‚0.03125# disk.dm-0.out‚29354.28076171875 disk.sda1.out‚0$ memory_percent‚92.10332234700522* proc.memory.percent‚11.829026540120442 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd2111.out‚0" proc.memory.rssMB‚ 2907.1015625Í`|0¡‚(:¥  ݾŠÓ·…ý³ network.recv‚513041% disk./.usageGB‚1260.8510208129883 disk.drbd1285.out‚0 disk.sda2.out‚ 233.01953125 disk.drbd1931.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚81.5 network.sent‚ 8.768353e+06 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚29355.02294921875 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.rbd1.out‚45.59375 disk.nvme0n1.in‚81.5 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚29355.02294921875* proc.memory.percent‚11.829026540120442 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0% proc.memory.availableMB‚ 1940.6875 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.sda.out‚ 233.01953125 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.51338957783547 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0$ memory_percent‚92.10332234700522 cpu‚25.317105332403834 disk.drbd2111.in‚0 proc.cpu.threads‚17j‚{OŸ¢‚(:” ݾŠÓá‰Ù³ gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.164' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.72133333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ÿÍž£‚(:“ 쾊Ó£ýÖ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.531' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.17699999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈË„ð/¤‚(:¤  쾊ÓãÖó³ network.sent‚ 8.768419e+06) disk./.usagePercent‚70.51338957783547 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1289.out‚0& disk.nvme0n1.out‚29355.47607421875 disk.dm-0.in‚81.5 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda.out‚ 233.31640625 disk.sda2.out‚ 233.31640625 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0# disk.dm-0.out‚29355.47607421875 disk.drbd2078.out‚0% proc.memory.availableMB‚ 1940.6875 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚81.5 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2907.1015625 cpu‚25.31696667164358 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚45.59375 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 network.recv‚513107% disk./.usageGB‚1260.8510208129883 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.829026540120442 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0$ memory_percent‚92.10332234700522 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1931.out‚0ÞÜ’ž¥‚(:“ û¾ŠÓÈôâ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚295.324' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.44133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ‰s5¦‚(:ª  û¾ŠÓËÞ´ disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0( proc.memory.availableMB‚ 1941.4453125 cpu‚25.316910287522 proc.cpu.threads‚17 network.sent‚ 8.768527e+06 network.recv‚513215 disk.dm-0.in‚81.5 disk.drbd1356.out‚0$ memory_percent‚92.10023880004883 disk.drbd1584.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.sda2.in‚0.03125 disk.rbd1.out‚ 45.7275390625 disk.nvme0n1.in‚81.5 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 233.58984375* proc.memory.percent‚11.829026540120442 disk.drbd1938.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51338957783547% disk./.usageGB‚1260.8510208129883 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 233.58984375 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚29355.78857421875 disk.drbd1552.in‚0 disk.drbd1817.in‚0# disk.dm-0.out‚29355.78857421875 disk.drbd1828.in‚0 disk.drbd1226.out‚0uZŽ2§- _wandbruntime‚9019  _runtime‚90192÷_²Ç¨Á & lr-AdamW/pg1‚8.672881156018331e-05 & lr-AdamW/pg2‚8.672881156018331e-05 trainer/global_step‚4949 % _timestamp‚1.7848482558616402e+09 _runtime‚9019.218917029 _step‚409n’Ì©Æ _step‚409 & lr-AdamW/pg1‚8.672881156018331e-05 & lr-AdamW/pg2‚8.672881156018331e-05 trainer/global_step‚4949 % _timestamp‚1.7848482558616402e+09 _runtime‚9019.218917029™÷®Cð2ª- _wandbruntime‚9019  _runtime‚9019pÿ®8(«ýO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88881577 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1423658592911120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848482558627188e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88891647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390026698311680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6313929868288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1429383273515008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6313981135360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5155422825690 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42941408 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1424220550240256 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1424220550240256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45940175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396340628179968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42941408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390027066402816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396341047538176 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1423658567483152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚410 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5154078214404 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5162723274752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15057798 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45950239 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1429383273515008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5164068710912 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28247313 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5164068710912 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396340628179968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42941408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390027066402816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396341047538176 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1423658567483152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚410 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5154078214404 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5162723274752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15057798 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45950239 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1429383273515008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5164068710912 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28247313 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0š|ëz;2­- _wandbruntime‚9020  _runtime‚9020ÉÆQm ®‡N UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5163745782784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5165091226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390105284677632 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88899026 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _runtime‚9020.700450119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396420467844096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88899026 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5155099016044 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45959335 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13192512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390105652768768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6315236523520 ;5DeviceStatsMonitor.on_train_batch_end/num_d]ؼfïevice_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45949257 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1429664576930304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1424499459530752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45959335 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚411 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42949775 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396420889292288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88909110 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1424499459530752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5163745782784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45949257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15060360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88909110 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28252867 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6315183166464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1429093838312338 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28253222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848482573428237e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1424499485704192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15060710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42949769 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88909110 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1424499459530752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5163745782784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45949257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15060360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88909110 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28252867 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6315183166464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1429093838312338 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28253222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848482573428237e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1424499485704192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15060710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42949769 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _runtime‚9020.700450119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396420467844096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88899026 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5155099016044 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45959335 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248›ƒÃ‰Ý2°- _wandbruntime‚9020  _runtime‚9020Â#*f ±„ 2loss/SRqqt2/assignment_loss‚0.5356603804191638 trainer/global_step‚4949 2loss/FBt2/assignment_loss‚0.023750940760386852 /loss/FBt1/detection_loss‚0.7874471722170711 epoch‚17 4loss/FBt1/assignment_loss‚2.2641472775308813e-05 1loss/SRqqt1/detection_loss‚0.9065182465848393 1loss/SRqqt2/detection_loss‚0.6650399319235519 %loss/total_loss‚8.895205731497745 $ _timestamp‚1.784848257377068e+09 _runtime‚9020.734297241 _step‚412 /loss/FRt1/detection_loss‚0.6170152777705038 /loss/FRt2/detection_loss‚1.6580693459349638 /loss/FBt2/detection_loss‚0.5379309307358322 0loss/FRt1/assignment_loss‚1.7899974991590524 0loss/FRt2/assignment_loss‚0.7879625599716854 1loss/SRqqt1/assignment_loss‚0.585790804547918ø³dL²‰ 1loss/SRqqt1/detection_loss‚0.9065182465848393 1loss/SRqqt2/detection_loss‚0.6650399319235519 %loss/total_loss‚8.895205731497745 $ _timestamp‚1.784848257377068e+09 _runtime‚9020.734297241 _step‚412 /loss/FRt1/detection_loss‚0.6170152777705038 /loss/FRt2/detection_loss‚1.6580693459349638 /loss/FBt2/detection_loss‚0.5379309307358322 0loss/FRt1/assignment_loss‚1.7899974991590524 0loss/FRt2/assignment_loss‚0.7879625599716854 1loss/SRqqt1/assignment_loss‚0.585790804547918 2loss/SRqqt2/assignment_loss‚0.5356603804191638 trainer/global_step‚4949 2loss/FBt2/assignment_loss‚0.023750940760386852 /loss/FBt1/detection_loss‚0.7874471722170711 epoch‚17 4loss/FBt1/assignment_loss‚2.2641472775308813e-05œ5Å5ž³‚(:“ Š¿ŠÓ¤ä³ gpu.0.gpu‚70.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚204.892' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.29733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç+ª'4´‚(:©  Š¿ŠÓŽß…´% disk./.usageGB‚1260.8510398864746 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51339064452468 disk.sda2.in‚0.03125 disk.dm-0.in‚81.5 disk.drbd1356.out‚0$ memory_percent‚92.10138320922852 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚81.5 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 45.7275390625* proc.memory.percent‚11.829026540120442 cpu‚25.316854523669765 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 233.91015625 disk.drbd1289.in‚0 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1941.1640625 network.sent‚ 8.769502e+06 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 233.91015625 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 network.recv‚514203 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0$ disk.nvme0n1.out‚29356.216796875 disk.sda1.in‚0 disk.drbd1931.in‚0! disk.dm-0.out‚29356.216796875ý,±žµ‚(:“ ™¿ŠÓ°ý±³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚274.477' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.49233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ’Ví/¶‚(:¤  ™¿ŠÓÔÀг disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 45.7900390625 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.nvme0n1.in‚81.5 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 234.359375 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 234.359375 disk.dm-0.in‚81.5 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0$ disk.nvme0n1.out‚29356.357421875 disk.drbd1226.out‚0 disk.drbd1510.in‚0$ memory_percent‚92.10138320922852 disk.drbd1938.in‚0! disk.dm-0.out‚29356.357421875" proc.memory.rssMB‚ 2907.1015625 cpu‚25.316801566421656 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0( proc.memory.availableMB‚ 1941.1640625 network.sent‚ 8.76961e+06 network.recv‚514311% disk./.usageGB‚1260.8505401611328 disk.sda.in‚0.03125 disk.drbd1938.out‚0) disk./.usagePercent‚70.51336269726752 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0±÷Œ“·‚(:ˆ ¨¿ŠÓšôƳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚175.395' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚58.465 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܧ5-¸‚(:¢  ¨¿ŠÓš§é³ disk.sda1.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.829026540120442 disk.sda1.in‚0 disk.dm-0.in‚81.5 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1941.10546875 disk.drbd1931.out‚0" disk.dm-0.out‚29356.7099609375 disk.drbd1584.in‚0 disk.rbd1.out‚ 45.7900390625 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚81.5 disk.drbd1510.in‚0 network.recv‚516135 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0$ memory_percent‚92.10162162780762 disk.drbd1817.out‚0% disk.nvme0n1.out‚29356.7099609375 disk.drbd1226.in‚0# disk./.usageGB‚1260.85054397583 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2907.1015625 proc.cpu.threads‚17 network.sent‚ 8.844172e+06 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51336291060537 disk.drbd1817.in‚0 disk.sda.out‚ 234.609375 disk.drbd1673.out‚0 disk.sda2.out‚ 234.609375 disk.drbd2078.out‚0 disk.drbd1226.out‚0 cpu‚25.31671773437 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.03125 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚06ƒ¡ô-¹‚(:¢  ·¿ŠÓ²ùϳ disk.drbd1931.in‚0 disk.sda.out‚ 234.90625 disk.drbd1828.in‚0 disk.drbd1285.out‚0$ memory_percent‚92.10138320922852 disk.drbd1931.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.sda2.in‚0.03125 disk.rbd1.out‚ 46.009765625 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.nvme0n1.in‚81.5 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd2041.out‚0% disk.nvme0n1.out‚29356.8974609375( proc.memory.availableMB‚ 1941.1640625 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.dm-0.in‚81.5 disk.drbd1552.in‚0 disk.sda2.out‚ 234.90625 disk.drbd1938.out‚0 cpu‚25.316609023824416 disk.drbd1673.in‚0" disk.dm-0.out‚29356.8974609375 disk.drbd2118.out‚0) disk./.usagePercent‚70.51333389665902 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 network.sent‚ 8.84428e+06 network.recv‚516243$ disk./.usageGB‚1260.850025177002„[YjŸº‚(:” ·¿ŠÓ…Þ®³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚305.749' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.91633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"N•”»‚(:‰ Æ¿ŠÓžµÈ³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.104' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L`Á7,¼‚(:¡  Æ¿ŠÓûç³ disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8500061035156 disk.drbd1931.out‚0 disk.dm-0.out‚ 29357.28125 disk.rbd1.in‚0 disk.drbd1285.in‚0$ memory_percent‚92.10073153177896 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125) disk./.usagePercent‚70.51333282996981 disk.drbd1965.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 network.sent‚ 8.906714e+06 disk.sda2.out‚ 235.21484375 network.recv‚518128 disk.dm-0.in‚81.5 disk.drbd2041.out‚0 disk.nvme0n1.out‚ 29357.28125 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚81.5 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 235.21484375 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1941.32421875 cpu‚25.316553239793354 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 46.009765625 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0þx“½‚(:ˆ Õ¿ŠÓÿ²ë³ gpu.0.gpu‚24.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚333.231' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚111.077 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 $К,¾‚(:¡  Õ¿ŠÓÒé´ disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1552.in‚0 disk.drbd1965.out‚0( proc.memory.availableMB‚ 1940.9921875 disk.sda1.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚81.5 disk.drbd1226.in‚0 disk.drbd2078.in‚0 cpu‚25.3164969820695 disk.drbd1226.out‚0 disk.drbd1931.out‚0" proc.memory.rssMB‚ 2907.1015625) disk./.usagePercent‚70.51333282996981 disk.drbd1584.out‚0 disk.sda2.out‚ 235.75390625 disk.drbd1828.in‚0$ memory_percent‚92.10208257039389 disk.sda.out‚ 235.75390625 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.dm-0.in‚81.5 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0 network.sent‚ 8.90678e+06% disk./.usageGB‚1260.8500061035156" disk.nvme0n1.out‚ 29357.4296875 network.recv‚518194 disk.rbd1.out‚ 46.072265625 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.dm-0.out‚ 29357.4296875 disk.drbd1584.in‚0Ž:\Ë2¿- _wandbruntime‚9106  _runtime‚9106¢qLåÇÀÁ _step‚413 & lr-AdamW/pg1‚8.613910250819999e-05 & lr-AdamW/pg2‚8.613910250819999e-05 trainer/global_step‚4999 % _timestamp‚1.7848483430640633e+09 _runtime‚9106.421299078˜ëªëÌÁÆ & lr-AdamW/pg2‚8.613910250819999e-05 trainer/global_step‚4999 % _timestamp‚1.7848483430640633e+09 _runtime‚9106.421299078 _step‚413 & lr-AdamW/pg1‚8.613910250819999e-05ý×ÍÈ2Â- _wandbruntime‚9106  _runtime‚9106ÇlÁyÃýO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46396127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5206526708498 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28526134 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1443379927596032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5215258267136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5205182097212 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5213912830976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1443381299205632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43359758 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1443379927596032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46406191 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393956384700416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43359752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0‘(Ä‚P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43359758 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1443379927596032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46406191 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393956384700416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43359752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46396127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5206526708498 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28526134 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1443379927596032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5215258267136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5205182097212 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5213912830976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1443381299205632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864žÖÇÀ2Å- _wandbruntime‚9107  _runtime‚9107Oz>ô 'ƇN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89783412 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1443661231011328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89773328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _step‚415 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46415287 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1443661231011328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5216280782336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5207547510250 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46415287 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6377870414336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89783412 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚359 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5206202898852 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1437877457945728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1438444950228992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚9107.904217364 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89773328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1438444924055552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43368125 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 % _timestamp‚1.7848483445462575e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1438444950228992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5214935339008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28532040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28531681 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13324512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1443085030883946 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1437877483373696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46405209 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400412841480704 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1443659859394560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13324507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15207528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚4999 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5216280782336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43368125 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5206202898852 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1437877457945728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1438444950228992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚9107.904217364 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89773328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1438444924055552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43368125 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 % _timestamp‚1.7848483445462575e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1438444950228992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5214935339008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28532040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28531681 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13324512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1443085030883946 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1437877483373696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46405209 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400412841480704 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1443659859394560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13324507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15207528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚4999 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5216280782336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43368125 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89783412 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1443661231011328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89773328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _step‚415 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46415287 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1443661231011328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5216280782336ŸT鋾2È- _wandbruntime‚9107  _runtime‚9107¥à Ƀ /loss/FRt1/detection_loss‚0.5312848904431143 /loss/FRt2/detection_loss‚1.4743883476015656 %loss/total_loss‚8.677404491002225 epoch‚17 % _timestamp‚1.7848483445809224e+09 _step‚416 2loss/FBt1/assignment_loss‚0.012988899778292762 1loss/SRqqt2/detection_loss‚0.6755988225958557 /loss/FBt1/detection_loss‚0.8157495094858336 trainer/global_step‚4999 0loss/FRt2/assignment_loss‚0.8235971178090212 2loss/SRqqt1/assignment_loss‚0.6347044869973179 2loss/SRqqt2/assignment_loss‚0.5314272601777601 1loss/SRqqt1/detection_loss‚0.9400421681175463 _runtime‚9107.938144755 /loss/FRt1/assignment_loss‚1.760919789883254 2loss/FBt2/assignment_loss‚0.027346037940172152 /loss/FBt2/detection_loss‚0.4493571601724917ø8éʈ 2loss/FBt1/assignment_loss‚0.012988899778292762 1loss/SRqqt2/detection_loss‚0.6755988225958557 /loss/FBt1/detection_loss‚0.8157495094858336 trainer/global_step‚4999 0loss/FRt2/assignment_loss‚0.8235971178090212 2loss/SRqqt1/assignment_loss‚0.6347044869973179 2loss/SRqqt2/assignment_loss‚0.5314272601777601 1loss/SRqqt1/detection_loss‚0.9400421681175463 _runtime‚9107.938144755 /loss/FRt1/assignment_loss‚1.760919789883254 2loss/FBt2/assignment_loss‚0.027346037940172152 /loss/FBt2/detection_loss‚0.4493571601724917 /loss/FRt1/detection_loss‚0.5312848904431143 /loss/FRt2/detection_loss‚1.4743883476015656 %loss/total_loss‚8.677404491002225 epoch‚17 % _timestamp‚1.7848483445809224e+09 _step‚416 Àį¤žË‚(:“ ä¿ŠÓ°¯Ä³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚263.776' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.92533333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ?vÚ3Ì‚(:¨  ä¿ŠÓžàè³ disk.drbd1931.in‚0# disk.dm-0.out‚29357.84130859375 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1940.07421875 proc.cpu.threads‚17( disk./.usagePercent‚70.5134403522416 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8519287109375 cpu‚25.31636238274642 disk.sda.out‚ 235.8984375 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.rbd1.out‚ 46.072265625 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2907.1015625* proc.memory.percent‚11.829026540120442 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 235.8984375 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1938.out‚0& disk.nvme0n1.out‚29357.84130859375 network.sent‚ 8.974781e+06 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚81.5 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚81.5# memory_percent‚92.1058177947998 network.recv‚521378 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 óåj”Í‚(:‰ ó¿ŠÓßæʳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.942' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.314 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÉ@7΂(:¬  ó¿ŠÓ€ãë³ disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 46.2060546875 disk.sda.out‚ 236.15234375 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚29358.06005859375 disk.drbd1938.in‚0# disk.dm-0.out‚29358.06005859375 disk.drbd1965.out‚0$ memory_percent‚92.10616747538248 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1939.98828125 disk.nvme0n1.in‚81.5 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 236.15234375 disk.dm-0.in‚81.5 network.recv‚521510% disk./.usageGB‚1260.8519096374512 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1828.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1356.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 cpu‚25.31630821093782( disk./.usagePercent‚70.5134392855524 disk.sda.in‚0.03125" proc.memory.rssMB‚ 2907.1015625 disk.rbd1.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 network.sent‚ 8.975584e+06ååžÏ‚(:“ ‚ÀŠÓÝòȳ gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚107.756' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.91866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žI¢m=Ђ(:²  ‚ÀŠÓàùé³ disk.drbd1817.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0% disk./.usageGB‚1260.8519287109375 disk.sda.in‚0.03125 disk.drbd1931.out‚0 cpu‚25.316254443404482 disk.dm-0.in‚ 83.52734375) proc.memory.availableMB‚ 1939.91015625 network.sent‚ 8.97565e+06( disk./.usagePercent‚70.5134403522416 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 236.6015625 disk.drbd2041.in‚0 network.recv‚521576 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0$ memory_percent‚92.10648536682129 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0# disk.nvme0n1.out‚29358.35546875 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.dm-0.out‚29358.35546875 disk.drbd1356.in‚0 disk.rbd1.out‚ 46.2060546875" proc.memory.rssMB‚ 2907.1015625 disk.sda.out‚ 236.6015625 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.in‚0 disk.drbd1584.in‚0ùš¹ žÑ‚(:“ ‘ÀŠÓ£¯¶³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.095' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.03166666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DàÖ;Ò‚(:°  ‘ÀŠÓµ±Ù³ disk.sda.out‚ 236.9140625 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1817.out‚0 disk.sda2.out‚ 236.9140625 disk.dm-0.out‚ 29359.015625 disk.drbd2078.in‚0 network.sent‚ 8.976176e+06 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51346659279602 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 network.recv‚522783 disk.drbd1673.out‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1939.79296875 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 cpu‚25.316201499062448 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0! disk.nvme0n1.out‚ 29359.015625 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1828.out‚0$ memory_percent‚92.10696220397949 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2907.1015625* proc.memory.percent‚11.829026540120442% disk./.usageGB‚1260.8523979187012 disk.rbd1.out‚ 46.2685546875¤#TÞžÓ‚(:“  ÀŠÓÈÁ³ gpu.0.gpu‚95.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.826' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.60866666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®²ØFÔ‚(:»   ÀŠÓô€â³ disk.dm-0.in‚ 83.52734375 disk.drbd1550.out‚0 disk.sda2.out‚ 237.24609375 network.sent‚ 8.986734e+06 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 cpu‚25.316500326158618 disk.drbd1584.out‚0 disk.drbd1289.out‚0 network.recv‚524702# disk.dm-0.out‚29359.85791015625 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8524055480957 disk.drbd1938.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.rbd1.out‚ 46.3623046875 disk.nvme0n1.in‚ 83.52734375 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.sda.out‚ 237.24609375 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚29359.85791015625 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 1939.76171875 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1931.in‚0$ memory_percent‚92.10708936055502 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1817.in‚0" proc.memory.rssMB‚ 2907.1015625( disk./.usagePercent‚70.5134670194717 disk.drbd2111.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1289.in‚0Ìð&uÕ‚(:’ ¯ÀŠÓ€Œà³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.6' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.53333333333335 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lRáºDÖ‚(:¹  ¯ÀŠÓÀöþ³ disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 network.recv‚526329 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8524055480957 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0( proc.memory.availableMB‚ 1940.0078125* proc.memory.percent‚11.829026540120442# disk.dm-0.out‚29360.14697265625 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0( disk./.usagePercent‚70.5134670194717" proc.memory.rssMB‚ 2907.1015625 cpu‚25.320562778299255 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚29360.14697265625 disk.sda1.out‚0# memory_percent‚92.1060880025228 network.sent‚ 8.987946e+06 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 237.54296875 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2111.out‚0 disk.drbd1931.out‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 46.5263671875 disk.sda2.out‚ 237.54296875 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd2118.out‚0m¨Sö2×- _wandbruntime‚9197  _runtime‚91974àÐØì0Øät 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7403532862663269 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3785969614982605 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8997966051101685 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4417031705379486 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7074988484382629 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20142242312431335 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8709522485733032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6839529871940613 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9926691651344299 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03465844690799713 +jet/accuracy_3_of_4‚0.09782741963863373 /particle/accuracy_4_of_4‚0.9798715710639954 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20813606679439545 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.64537513256073 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9063828587532043 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18156611919403076 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2315077781677246 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6815846562385559 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9985784888267517 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03184843808412552 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6294870972633362 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9177736043930054 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3802662491798401 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8489390015602112 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2809523940086365 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 +particle/sensitivity‚0.6363540291786194 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7192307710647583 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3228086531162262 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24678678810596466 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.64537513256073 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8709522485733032 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.73322993516922 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1988207846879959 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 particle/accuracy_2_of_4‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18215426802635193 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11712954193353653 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13113661110401154 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035666532814502716 particle/accuracy_1_of_1‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19572778046131134 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7506313323974609 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 5 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.79058438539505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27679768204689026 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5934925675392151 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.694116473197937 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4520663917064667 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6412066221237183 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.035285834223032 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21252615749835968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4593164324760437 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16564181447029114 *jet/accuracy_2_of_4‚0.8343881964683533 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.738901674747467 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.634429931640625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3127969205379486 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9965197443962097 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5857346653938293 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4502416253089905 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17521922290325165 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27800390124320984 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21184714138507843 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023720739409327507 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21568281948566437 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6571844816207886 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7455063462257385 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43294578790664673 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.45384615659713745 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6270161271095276 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879184365272522 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017334314063191414 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 )loss/val_total_loss‚8.388643364658707 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9982362985610962 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 _step‚417 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 % _timestamp‚1.7848484338922641e+09 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7929292917251587 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22723206877708435 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23011426627635956 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 *jet/accuracy_1_of_3‚0.9320657849311829 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927783012390137 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47727271914482117 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2279866635799408 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023720739409327507 trainer/global_step‚5039 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7066453695297241 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9908443689346313 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 +jet/accuracy_2_of_2‚0.14818140864372253 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6430791616439819 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8866353034973145 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029764264822006226 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6443371176719666 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893012046813965 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48338931798934937 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 epoch‚17 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21568281948566437 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9975091218948364 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.785977303981781 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5907951593399048 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6684507131576538 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7663885354995728 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7791310548782349 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6631512641906738 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06907475739717484 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6485997438430786 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8928571343421936 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7166863083839417 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19569964706897736 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9989533424377441 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7842068076133728 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8066564202308655 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5630263686180115 *jet/accuracy_1_of_4‚0.9707711338996887 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.697359561920166 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8487505316734314 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_3‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7385033369064331 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02817368507385254 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 *jet/accuracy_1_of_2‚0.7716647386550903 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9972156882286072 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.36478278040885925 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19637489318847656 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21540537476539612 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034106ÀÉD~ 895327568054 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04174771159887314 _runtime‚9197.250359453 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 (particle/accuracy‚0.8484468460083008 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5966433882713318 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5935472846031189 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6610410809516907 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7466840744018555 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17714610695838928 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7844424843788147 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 +jet/accuracy_3_of_3‚0.06394017487764359 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7692099213600159 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7350967526435852 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7346596121788025 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2001338005065918 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3635854423046112 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10085014253854752 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7431713938713074 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11414630711078644 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6434910893440247 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.805659830570221 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9963346123695374t%àyo:Ùét 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.73322993516922 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1988207846879959 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 particle/accuracy_2_of_4‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18215426802635193 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11712954193353653 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13113661110401154 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035666532814502716 particle/accuracy_1_of_1‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19572778046131134 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7506313323974609 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 5 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.79058438539505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27679768204689026 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5934925675392151 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.694116473197937 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4520663917064667 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6412066221237183 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.035285834223032 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21252615749835968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4593164324760437 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16564181447029114 *jet/accuracy_2_of_4‚0.8343881964683533 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.738901674747467 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.634429931640625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3127969205379486 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9965197443962097 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5857346653938293 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4502416253089905 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17521922290325165 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27800390124320984 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21184714138507843 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023720739409327507 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21568281948566437 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6571844816207886 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7455063462257385 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43294578790664673 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.45384615659713745 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6270161271095276 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879184365272522 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017334314063191414 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 )loss/val_total_loss‚8.388643364658707 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9982362985610962 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 _step‚417 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 % _timestamp‚1.7848484338922641e+09 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7929292917251587 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22723206877708435 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23011426627635956 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 *jet/accuracy_1_of_3‚0.9320657849311829 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927783012390137 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47727271914482117 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2279866635799408 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023720739409327507 trainer/global_step‚5039 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7066453695297241 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9908443689346313 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 +jet/accuracy_2_of_2‚0.14818140864372253 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6430791616439819 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8866353034973145 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029764264822006226 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6443371176719666 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893012046813965 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48338931798934937 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 epoch‚17 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21568281948566437 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9975091218948364 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.785977303981781 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5907951593399048 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6684507131576538 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7663885354995728 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7791310548782349 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6631512641906738 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06907475739717484 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6485997438430786 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8928571343421936 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7166863083839417 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19569964706897736 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9989533424377441 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7842068076133728 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8066564202308655 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5630263686180115 *jet/accuracy_1_of_4‚0.9707711338996887 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.697359561920166 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8487505316734314 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_3‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7385033369064331 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02817368507385254 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 *jet/accuracy_1_of_2‚0.7716647386550903 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9972156882286072 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.36478278040885925 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19637489318847656 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21540537476539612 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034106895327568054 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04174771159887314 _runtime‚9197.250359453 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 (particle/accuracy‚0.8484468460083008 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5966433882713318 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5935472846031189 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6610410809516907 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7466840744018555 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17714610695838928 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7844424843788147 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 +jet/accuracy_3_of_3‚0.06394017487764359 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7692099213600159 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7350967526435852 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7346596121788025 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2001338005065918 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3635854423046112 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10085014253854752 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7431713938713074 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11414630711078644 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6434910893440247 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.805659830570221 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9963346123695374 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7403532862663269 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3785969614982605 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8997966051101685 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4417031705379486 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7074988484382629 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20142242312431335 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8709522485733032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6839529871940613 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9926691651344299 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03465844690799713 +jet/accuracy_3_of_4‚0.09782741963863373 /particle/accuracy_4_of_4‚0.9798715710639954 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20813606679439545 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.64537513256073 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9063828587532043 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18156611919403076 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2315077781677246 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6815846562385559 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9985784888267517 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03184843808412552 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6294870972633362 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9177736043930054 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3802662491798401 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8489390015602112 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2809523940086365 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 +particle/sensitivity‚0.6363540291786194 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7192307710647583 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3228086531162262 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24678678810596466 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.64537513256073 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8709522485733032¡YÃ:}EÚ‚(:º  ¾ÀŠÓ–ãý³ disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 237.82421875" proc.memory.rssMB‚ 2907.1015625 cpu‚25.321259671822766 proc.cpu.threads‚17 disk.rbd1.out‚ 46.5263671875 disk.drbd1289.in‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.51346979286363 network.sent‚ 8.99027e+06 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1939.67578125* proc.memory.percent‚11.829026540120442 disk.drbd1828.in‚0 disk.sda2.out‚ 237.82421875 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0% disk./.usageGB‚1260.8524551391602 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.sda.in‚0.03125# disk.dm-0.out‚29360.40673828125 disk.drbd1673.out‚0 disk.drbd2118.out‚0# memory_percent‚92.1074390411377 disk.drbd1931.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 network.recv‚526503 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.nvme0n1.in‚ 83.52734375& disk.nvme0n1.out‚29360.40673828125 disk.drbd1552.out‚0 disk.drbd1673.in‚0ðçžÛ‚(:“ ¾ÀŠÓ¢ºß³ gpu.0.gpu‚52.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚289.264' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.42133333333334 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¸‚2Ü- _wandbruntime‚9213  _runtime‚9213nãÆÝÀ trainer/global_step‚5049 $ _timestamp‚1.784848449835876e+09 _runtime‚9213.193110774 _step‚418 & lr-AdamW/pg1‚8.554577905910327e-05 & lr-AdamW/pg2‚8.554577905910327e-05ªDËÞÅ _runtime‚9213.193110774 _step‚418 & lr-AdamW/pg1‚8.554577905910327e-05 & lr-AdamW/pg2‚8.554577905910327e-05 trainer/global_step‚5049 $ _timestamp‚1.784848449835876e+09¢Tz!È2ß- _wandbruntime‚9213  _runtime‚9213ñ§ï™(àüO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5272066259968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397934702511104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1459942876838400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6445950341120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1459944248448000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1454672156014592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚9213.194474156 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402580992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚419 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1459942876838400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28838540 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15377776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15377507 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5263239430615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1454672182188032 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1454097771439888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90776990 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1459359640831249 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397934334419968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28838266 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13460764 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46909450 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43867540Ä0‡Tó áP E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90766920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13460764 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46909450 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43867540 ^LDeviceStatsMonitor.on_Áî;ltrain_batch_start/allocated_bytes.small_pool.allocated‚ 5272066259968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397934702511104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1459942876838400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6445950341120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1459944248448000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1454672156014592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚9213.194474156 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402580992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚419 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1459942876838400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28838540 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15377776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15377507 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5263239430615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1454672182188032 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1454097771439888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90776990 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1459359640831249 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397934334419968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28838266 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5049 $ _timestamp‚1.784848451316096e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1460224180253696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5264260232367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6447267056640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1454951091478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5262915620969 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46908468 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46908468 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1460222808636928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15380809 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46918546 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1454376547785696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28844213 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28843949 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404460153359872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398013288877056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚9214.673696835 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15380550 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5271743331840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1454951065305088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13463399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43875907 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5273088775168 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1454951065305088 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13463404 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90794453 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6447232573952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1454951091478528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005»ËQC'ä‹N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15380809 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46918546 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1454376547785696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28844213 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28843949 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404460153359872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398013288877056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚9214.673696835 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15380550 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5271743331840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1454951065305088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13463399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43875907 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5273088775168 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1454951065305088 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13463404 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90794453 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6447232573952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1454951091478528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1459640833446031 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90784369 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5049 $ _timestamp‚1.784848451316096e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1460224180253696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5264260232367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6447267056640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1454951091478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5262915620969 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46908468 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46908468 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1460222808636928¤Û–¡2å- _wandbruntime‚9214  _runtime‚9214÷“žæÿ $ _timestamp‚1.784848451352315e+09 0loss/FRt1/assignment_loss‚1.7484746966814606 2loss/SRqqt2/assignment_loss‚0.6182686323366458 /loss/FRt2/detection_loss‚1.4820280517485374 _runtime‚9214.709515785 /loss/FBt2/detection_loss‚0.3593868392109807 _step‚421 0loss/FRt2/assignment_loss‚0.7072807057100592 1loss/FBt2/assignment_loss‚0.03965114270499585 1loss/SRqqt2/detection_loss‚0.6963457376214163 /loss/FBt1/detection_loss‚0.3689136943100272 epoch‚18 2loss/SRqqt1/assignment_loss‚0.5441561167341371 3loss/FBt1/assignment_loss‚0.0015395449397839751 ,loss/FRt1/detection_loss‚0.5542216119995 0loss/SRqqt1/detection_loss‚0.994372806271898 %loss/total_loss‚8.114639580269442 trainer/global_step‚5049|ZRÏ ç„ 2loss/SRqqt1/assignment_loss‚0.5441561167341371 3loss/FBt1/assignment_loss‚0.0015395449397839751 ,loss/FRt1/detection_loss‚0.5542216119995 0loss/SRqqt1/detection_loss‚0.994372806271898 %loss/total_loss‚8.114639580269442 trainer/global_step‚5049 $ _timestamp‚1.784848451352315e+09 0loss/FRt1/assignment_loss‚1.7484746966814606 2loss/SRqqt2/assignment_loss‚0.6182686323366458 /loss/FRt2/detection_loss‚1.4820280517485374 _runtime‚9214.709515785 /loss/FBt2/detection_loss‚0.3593868392109807 _step‚421 0loss/FRt2/assignment_loss‚0.7072807057100592 1loss/FBt2/assignment_loss‚0.03965114270499585 1loss/SRqqt2/detection_loss‚0.6963457376214163 /loss/FBt1/detection_loss‚0.3689136943100272 epoch‚18¥ðüç¯Eè‚(:º  ÍÀŠÓÀïì³ disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0( proc.memory.availableMB‚ 1939.8828125 disk.drbd2118.out‚0 disk.drbd1931.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.dm-0.in‚ 83.52734375 disk.drbd1550.out‚0 disk.sda1.in‚0 network.sent‚ 8.990336e+06 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0$ memory_percent‚92.10659662882486 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 cpu‚25.32114496584234 disk.drbd2111.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 46.7685546875* proc.memory.percent‚11.829026540120442 network.recv‚526569 disk.sda.out‚ 238.37109375 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1584.in‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51347597966102 disk.drbd1817.out‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚29360.75048828125 disk.drbd1510.out‚0 disk.sda1.out‚0% disk./.usageGB‚1260.8525657653809 disk.drbd1552.in‚0 disk.sda2.out‚ 238.37109375# disk.dm-0.out‚29360.75048828125 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1356.in‚0 disk.drbd1673.in‚0`ÂÝ žé‚(:“ ÍÀŠÓÿ˳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.968' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.98933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®Kš“ê‚(:ˆ ÜÀŠÓóàÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.754' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.918 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ƒa™ë‚(:¹  ÜÀŠÓ°€ó³" proc.memory.rssMB‚ 2907.1015625 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1289.out‚0 disk.drbd1965.out‚0( proc.memory.availableMB‚ 1939.8828125 disk.drbd2078.out‚0 disk.drbd2118.in‚0# disk.dm-0.out‚29362.03173828125) disk./.usagePercent‚70.51347704635023 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1931.in‚0$ memory_percent‚92.10659662882486 disk.drbd1289.in‚0 disk.drbd1938.in‚0 network.sent‚ 8.990862e+06 disk.rbd1.out‚ 46.9482421875& disk.nvme0n1.out‚29362.03173828125 disk.drbd1931.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 diMcQj«sk.sda1.in‚0 disk.drbd1673.in‚0 cpu‚25.321137453596137 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 83.52734375 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 238.9609375 network.recv‚527629 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 238.9609375% disk./.usageGB‚1260.8525848388672 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0·m 2Ÿì‚(:” ëÀŠÓ×ãÞ³ gpu.0.gpu‚78.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.582' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.193999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê›§þ;í‚(:°  ëÀŠÓÚ¦ü³ disk.dm-0.in‚ 83.52734375 disk.drbd1510.in‚0 cpu‚25.321102904466862) proc.memory.availableMB‚ 1939.84765625) disk./.usagePercent‚70.51347277959341 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda2.out‚239.25 disk.drbd1931.out‚0$ memory_percent‚92.10673967997232 network.recv‚527695 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1965.out‚0 disk.sda.in‚0.03125 network.sent‚ 8.990928e+06 disk.drbd1938.in‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚29362.28173828125 disk.drbd1226.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda.out‚239.25 disk.drbd1673.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8525085449219 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 47.1123046875 disk.drbd1289.out‚0 proc.cpu.threads‚17* proc.memory.percent‚11.829026540120442 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0# disk.dm-0.out‚29362.28173828125 disk.drbd1552.out‚0 disk.drbd1289.in‚0£êÛ9žî‚(:“ úÀŠÓà°°³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚280.594' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.53133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_€`VCï‚(:¸  úÀŠÓ˨ϳ disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚29363.42333984375 disk.drbd1550.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚29363.42333984375 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 239.6484375 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0) disk./.usagePercent‚70.51347277959341 disk.drbd1356.in‚0 disk.drbd1510.out‚0 cpu‚25.32099027774787 disk.drbd1289.in‚0 disk.rbd1.out‚ 47.1982421875$ memory_percent‚92.10672378540039 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1828.out‚0 network.recv‚527761% disk./.usageGB‚1260.8525085449219 disk.drbd1938.out‚0 disk.drbd1965.in‚0( proc.memory.availableMB‚ 1939.8515625 network.sent‚ 8.990994e+06 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 239.6484375 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd2118.in‚0 disk.drbd2041.out‚0ûÐuFð‚(:»  ‰ÁŠÓ¨ŠÖ³ disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 47.2607421875 disk.drbd1673.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 240.30859375 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1939.8515625" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 8.991144e+06% disk./.usageGB‚1260.8525085449219 disk.drbd2078.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51347277959341 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1938.out‚0& disk.nvme0n1.out‚29363.58349609375 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0# disk.dm-0.out‚29363.58349609375 disk.drbd1828.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.sda.out‚ 240.30859375 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1510.in‚0 disk.dm-0.in‚ 83.52734375 cpu‚25.320902528902575 disk.sda1.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0$ memory_percent‚92.10672378540039 disk.drbd2111.out‚0 network.recv‚527911 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0í¢7žñ‚(:“ ‰ÁŠÓ•ö´³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.576' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.52533333333335 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê×Úžò‚(:“ ˜ÁŠÓ´¾³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚120.302' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.10066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â7cCó‚(:¸  ˜ÁŠÓäŸß³ disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚29364.10400390625 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0) disk./.usagePercent‚70.51345891263375 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0& proc.memory.availableMB‚ 1939.84375* proc.memory.percent‚11.829026540120442 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 47.2607421875 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚29364.10400390625 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 83.52734375% disk./.usageGB‚1260.8522605895996 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 240.58203125 disk.sda2.out‚ 240.58203125 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.10675557454428 disk.drbd1552.out‚0 disk.drbd1550.in‚0 network.recv‚532043 disk.drbd2078.in‚0" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 9.082322e+06 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.sda.in‚0.03125 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 83.52734375 cpu‚25.32081498510676Ý~Úo2ô- _wandbruntime‚9300  _runtime‚9300%iÔ4ÆõÀ & lr-AdamW/pg1‚8.494891898400359e-05 & lr-AdamW/pg2‚8.494891898400359e-05 trainer/global_step‚5099 $ _timestamp‚1.784848536869335e+09 _runtime‚9300.226588739 _step‚422ÞùÚëËöÅ & lr-AdamW/pg1‚8.494891898400359e-05 & lr-AdamW/pg2‚8.494891898400359e-05 trainer/global_step‚5099 $ _timestamp‚1.784848536869335e+09 _runtime‚9300.226588739 _step‚422¦© C%2÷- _wandbruntime‚9300  _runtime‚9300†7((øýO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1468037834700720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44285890 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6509288288256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1473940902529024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47355338 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1468617646712832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5312998702137 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29119910 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1473350833402857 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91641222 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29119560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401863652717568 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1473939530919424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1468617620539392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9300.228103025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15527146 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47365402 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1473940902529024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6509333263872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44285884 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408372941005824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401864020808704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5314343313423Ás(ù‚P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6509333263872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44285884 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408372941005824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401864020808704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5314343313423 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1468037834700720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44285890 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6509288288256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1473940902529024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47355338 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1468617646712832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5312998702137 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29119910 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1473350833402857 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91641222 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29119560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401863652717568 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1473939530919424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1468617620539392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9300.228103025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15527146 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47365402 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1473940902529024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0§ûöoI2ú- _wandbruntime‚9301  _runtime‚9301ÐÍLÄ, û†N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 _runtime‚ 9301.70956642 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1474220834334720 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1468896556003328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47364420 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401942607174656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6510529551360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91668755 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44294251 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408452768634880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13595404 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚408453194277376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91668755 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47374498 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44294257 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1468316661902464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1468896529829888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13595399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15530101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5314019503777 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5324278331392 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5324278331392 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1468896556003328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29125152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91658671 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1468896529829888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401942239083520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1474220834334720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15529753 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5322932888064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚348 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47364420 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44294257 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5322932888064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5315364115175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91658671 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1473632026017639 % _timestamp‚1.7848485383519132e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1468896529829888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13595399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15530101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5314019503777 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5324278331392 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5324278331392 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1468896556003328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29125152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91658671 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1468896529829888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401942239083520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1474220834334720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15529753 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5322932888064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚348 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47364420 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44294257 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5322932888064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5315364115175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91658671 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1473632026017639 % _timestamp‚1.7848485383519132e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1468896556003328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47364420 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401942607174656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6510529551360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91668755 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44294251 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408452768634880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¨7S2ý- _wandbruntime‚9301  _runtime‚9301¯ãëöþ 0loss/FRt2/assignment_loss‚0.7021352851417507 2loss/FBt1/assignment_loss‚0.024836877672624037 0loss/FBt2/assignment_loss‚0.0447231150217802 1loss/SRqqt2/detection_loss‚0.6189640373410287 /loss/FBt1/detection_loss‚0.6020611556442944 %loss/total_loss‚8.333852416060244 epoch‚18 2loss/SRqqt1/assignment_loss‚0.5481309022262343 % _timestamp‚1.7848485383845935e+09 _step‚425 .loss/FRt2/detection_loss‚1.449173092648337 1loss/SRqqt1/detection_loss‚0.9481976225848433 /loss/FBt2/detection_loss‚0.4916682125135724 trainer/global_step‚5099 2loss/SRqqt2/assignment_loss‚0.5245144767056206 /loss/FRt1/detection_loss‚0.5586527836552937 _runtime‚9301.741808636 0loss/FRt1/assignment_loss‚1.8207948549048645( Æ ÿ† 0loss/FRt1/assignment_loss‚1.8207948549048645 0loss/FRt2/assignment_loss‚0.7021352851417507 2loss/FBt1/assignment_loss‚0.024836877672624037 0loss/FBt2/assignment_loss‚0.0447231150217802 1loss/SRqqt2/detection_loss‚0.6189640373410287 /loss/FBt1/detection_loss‚0.6020611556442944 %loss/total_loss‚8.333852416060244 epoch‚18 2loss/SRqqt1/assignment_loss‚0.5481309022262343 % _timestamp‚1.7848485383845935e+09 _step‚425 .loss/FRt2/detection_loss‚1.449173092648337 1loss/SRqqt1/detection_loss‚0.9481976225848433 /loss/FBt2/detection_loss‚0.4916682125135724 trainer/global_step‚5099 2loss/SRqqt2/assignment_loss‚0.5245144767056206 /loss/FRt1/detection_loss‚0.5586527836552937 _runtime‚9301.741808636©(§ž€‚(:“ §ÁŠÓ„æÀ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.471' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.15700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:î,MD‚(:¹  §ÁŠÓˆ™á³% disk./.usageGB‚1260.8523406982422 disk.drbd2111.out‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0& disk.nvme0n1.out‚29364.33056640625 disk.sda1.in‚0 network.recv‚532175 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 cpu‚25.320781471012655 network.sent‚ 9.083126e+06( disk./.usagePercent‚70.5134633927284 disk.dm-0.in‚ 83.52734375 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.10689862569174 disk.drbd1673.in‚0# disk.dm-0.out‚29364.33056640625 disk.drbd2078.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 240.87890625 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0) proc.memory.availableMB‚ 1939.80859375" proc.memory.rssMB‚ 2907.1015625 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 47.39453125 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.sda.out‚ 240.87890625 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0r¢·Ÿ‚‚(:” ¶ÁŠÓçšÉ³ gpu.0.gpu‚36.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚303.828' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.27599999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0` P9ƒ‚(:®  ¶ÁŠÓŠ¯é³ disk.drbd1938.out‚0# disk.dm-0.out‚29364.70654296875 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.829026540120442 network.sent‚ 9.083234e+06 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1939.80859375% disk./.usageGB‚1260.8523406982422 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 cpu‚25.320721127798$ memory_percent‚92.10689862569174 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.out‚241.1875 disk.drbd1965.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚29364.70654296875 disk.drbd1584.in‚0 disk.sda.in‚0.03125 proc.cpu.threads‚17 network.recv‚532283 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 47.39453125( disk./.usagePercent‚70.5134633927284 disk.dm-0.in‚ 83.52734375 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚241.1875 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1510.out‚0" proc.memory.rssMB‚ 2907.1015625 :hÛž„‚(:“ ÅÁŠÓÖ𶳠gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚272.833' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.94433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥þ´QB…‚(:·  ÅÁŠÓ¬ÒÕ³ disk.drbd1510.in‚0 disk.sda.out‚ 241.5546875 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 9.083611e+06 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 83.52734375 disk.rbd1.out‚ 47.45703125 proc.cpu.threads‚17 network.recv‚533336 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1510.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 241.5546875# disk.dm-0.out‚29365.34716796875 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 cpu‚25.32058222185791 disk.drbd1673.in‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51351736720214 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚29365.34716796875 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.8533058166504 disk.drbd1673.out‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1939.80859375$ memory_percent‚92.10689862569174 disk.drbd2118.out‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.829026540120442ÁÍÜv“†‚(:ˆ ÔÁŠÓµÿº³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚274.824' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄZ±@‡‚(:µ  ÔÁŠÓžÄÙ³) proc.memory.availableMB‚ 1940.52734375 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.829026540120442 cpu‚25.32052257503113 network.sent‚ 9.083719e+06 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 241.8203125 disk.rbd1.out‚ 47.45703125 disk.drbd1965.out‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17" disk.dm-0.out‚29365.6669921875 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd2118.in‚0) disk./.usagePercent‚70.51351736720214 disk.drbd1285.out‚0% disk.nvme0n1.out‚29365.6669921875 disk.drbd1226.in‚0$ memory_percent‚92.10397402445474 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 network.recv‚533444 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8533058166504 disk.drbd1552.in‚0 disk.dm-0.in‚ 83.52734375 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.sda.out‚ 241.8203125 disk.drbd2078.out‚0‘…Ÿˆ‚(:” ãÁŠÓøÄÞ³ gpu.0.gpu‚36.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚308.876' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.95866666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ГYA‰‚(:¶  ãÁŠÓ•þ³ disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0% disk.nvme0n1.out‚29365.9873046875 disk.drbd1510.out‚0 disk.sda.in‚0.03125" proc.memory.rssMB‚ 2907.1015625 disk.sda2.in‚0.03125 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 network.recv‚535044% disk./.usageGB‚1260.8533248901367 disk.drbd1226.out‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 242.0859375$ memory_percent‚92.10313161214194 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 47.5908203125 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51351843389135 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.829026540120442 cpu‚25.320461674937178 disk.dm-0.in‚ 83.52734375" disk.dm-0.out‚29365.9873046875 disk.nvme0n1.in‚ 83.52734375 disk.drbd1285.in‚0' proc.memory.availableMB‚ 1940.734375 disk.drbd1817.out‚0 disk.sda2.out‚ 242.0859375 network.sent‚ 9.148114e+06ñ¹R 2Š- _wandbruntime‚9386  _runtime‚9386©Z‰Ç‹Á _runtime‚9386.964516055 _step‚426 & lr-AdamW/pg1‚8.434860051758206e-05 & lr-AdamW/pg2‚8.434860051758206e-05 trainer/global_step‚5149 % _timestamp‚1.7848486236072903e+09³UÓØÌŒÆ & lr-AdamW/pg2‚8.434860051758206e-05 trainer/global_step‚5149 % _timestamp‚1.7848486236072903e+09 _runtime‚9386.964516055 _step‚426 & lr-AdamW/pg1‚8.434860051758206e-05ª x8:2- _wandbruntime‚9386  _runtime‚9386ÄQΑú ŽýO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47821354 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6571837237760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44704234 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47811290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚9386.968916385 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1482563111237632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44704240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1481977923389520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1487936185000448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5373099936256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15675072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6571781776384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1482563111237632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29399836 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405793339106304 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 % _timestamp‚1.7848486236083195e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5374445372416 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1487937556610048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224.ïâ›(‚P L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47811290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚9386.968916385 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1482563111237632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44704240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1481977923389520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1487936185000448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5373099936256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15675072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6571781776384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1482563111237632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29399836 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405793339106304 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 % _timestamp‚1.7848486236083195e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5374445372416 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1487937556610048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47821354 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6571837237760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44704234«•6‹"2- _wandbruntime‚9388  _runtime‚93887‘ó! '‘‡N _step‚428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412444584385536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29405067 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1482842020528128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5374122444288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1488217488415744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92543057 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44712607 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412445014222336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1487623218589247 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1482841994354688 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44712601 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47830450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15678035 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44712601 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92543057 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚9388.439609206 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5366467997983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92532973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405871557381120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6573088750080 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1482256725163296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29405067 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1482842020528128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5374122444288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1488217488415744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92543057 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44712607 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412445014222336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1487623218589247 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1482841994354688 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44712601 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47830450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15678035 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44712601 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92543057 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚9388.439609206 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5366467997983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92532973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405871557381120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6573088750080 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1482256725163296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1496508575762432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 % _timestamp‚1.7848487103633807e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1501934210691072 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1501932839081472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5425634928640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15822899 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5424289492480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1501934210691072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15822531 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1501333218546073 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29679663 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1495918037506288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 9473.72103913 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1496508549588992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1501932839081472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409722657403904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1496508549588992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5425634928640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1495918012078320 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5424289492480 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚431 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13856759 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416356849879552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45122590 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48277306 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93399896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416356434715648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 trainer/global_step‚5199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45122584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48267242 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29679290 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5415206467753 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1496508575762432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48277306 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976éY5]_§P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13856759 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416356849879552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45122590 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48277306 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93399896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416356434715648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 trainer/global_step‚5199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45122584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48267242 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29679290 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5415206467753 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1496508575762432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48277306 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48267242 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45122584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6634145402880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/acy¦Ÿ¨tive.large_pool.peak‚1653 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1496508575762432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 % _timestamp‚1.7848487103633807e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1501934210691072 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1501932839081472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5425634928640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15822899 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5424289492480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1501934210691072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15822531 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1501333218546073 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29679663 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1495918037506288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 9473.72103913 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1496508549588992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1501932839081472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409722657403904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1496508549588992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5425634928640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1495918012078320 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5424289492480 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚431¯yø&’2¨- _wandbruntime‚9475  _runtime‚9475¨6_™ '©‡N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93417359 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409801243769856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93417359 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1501614411160855 % _timestamp‚1.7848487118411522e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93407275 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1496787485052928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚9475.198774845 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15825516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29684915 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5425312000512 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15825872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5417571880791 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚432 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1502214142496768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13859404 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45130957 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5425312000512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1496787458879488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93407275 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45130951 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48276324 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48286402 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1496787458879488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5426657443840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6635453909504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1502214142496768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48276324 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112ìiºÆ'ªŒN REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45130957 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5425312000512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1496787458879488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93407275 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45130951 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48276324 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48286402 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1496787458879488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5426657443840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6635453909504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1502214142496768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48276324 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93417359 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409801243769856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93417359 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1501614411160855 % _timestamp‚1.7848487118411522e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93407275 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1496787485052928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚9475.198774845 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15825516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29684915 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5425312000512 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15825872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5417571880791 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚432 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1502214142496768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13859404 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0° ¿Ø2«- _wandbruntime‚9475  _runtime‚9475åG‹¬ 2loss/FBt1/assignment_loss‚0.006688571613962669 2loss/FBt2/assignment_loss‚0.030322762157058303 0loss/SRqqt1/detection_loss‚0.941827031728396 _runtime‚9475.233413154 2loss/SRqqt2/assignment_loss‚0.5246110784669346 /loss/FRt1/detection_loss‚0.5566022361972608 /loss/FBt1/detection_loss‚0.6777251062519623 trainer/global_step‚5199 #loss/total_loss‚8.5653797983999 0loss/FRt1/assignment_loss‚1.7928958382662825 2loss/SRqqt1/assignment_loss‚0.5685228783359424 /loss/FRt2/detection_loss‚1.5619464711677553 /loss/FBt2/detection_loss‚0.5243146553957097 % _timestamp‚1.7848487118762112e+09 1loss/SRqqt2/detection_loss‚0.6204101444159336 _step‚433 epoch‚18 0loss/FRt2/assignment_loss‚0.7595130244027013©gÉé ­† 0loss/FRt1/assignment_loss‚1.7928958382662825 2loss/SRqqt1/assignment_loss‚0.5685228783359424 /loss/FRt2/detection_loss‚1.5619464711677553 /loss/FBt2/detection_loss‚0.5243146553957097 % _timestamp‚1.7848487118762112e+09 1loss/SRqqt2/detection_loss‚0.6204101444159336 _step‚433 epoch‚18 0loss/FRt2/assignment_loss‚0.7595130244027013 2loss/FBt1/assignment_loss‚0.006688571613962669 2loss/FBt2/assignment_loss‚0.030322762157058303 0loss/SRqqt1/detection_loss‚0.941827031728396 _runtime‚9475.233413154 2loss/SRqqt2/assignment_loss‚0.5246110784669346 /loss/FRt1/detection_loss‚0.5566022361972608 /loss/FBt1/detection_loss‚0.6777251062519623 trainer/global_step‚5199 #loss/total_loss‚8.5653797983999±I½,Š“®‚(:ˆ ÌŠÓëÒdz gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.157' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.719 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0] ˆr?¯‚(:´  ÌŠÓ¦ªä³ disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51352632739146 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 47.9736328125 proc.cpu.threads‚17 network.recv‚544969 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1510.out‚0 disk.sda2.out‚ 244.9140625 disk.drbd1828.in‚0$ memory_percent‚92.10624694824219% disk./.usageGB‚1260.8534660339355 disk.drbd1550.in‚0 disk.dm-0.in‚ 83.52734375" disk.dm-0.out‚29367.9443359375 disk.drbd1965.out‚0 network.sent‚ 9.267653e+06 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0& proc.memory.availableMB‚ 1939.96875 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 244.9140625 disk.nvme0n1.in‚ 83.52734375 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0% disk.nvme0n1.out‚29367.9443359375" proc.memory.rssMB‚ 2907.1015625 disk.drbd2078.in‚0 cpu‚25.31994791354538 disk.drbd1226.in‚0póßU“°‚(:ˆ ÛŠÓŒ¼±³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.439' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.813 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö ‘¼B±‚(:·  ÛŠÓ¶×г disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0# memory_percent‚92.1072800954183 disk.drbd1817.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.rbd1.out‚ 48.2314453125 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚29368.1201171875( disk./.usagePercent‚70.5135301674726% disk./.usageGB‚1260.8535346984863 disk.drbd1510.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.out‚0 network.sent‚ 9.32738e+06 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 network.recv‚546556 disk.sda.out‚ 245.15234375 disk.drbd1285.out‚0* proc.memory.percent‚11.829026540120442 cpu‚25.319864882187325 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0" disk.dm-0.out‚29368.1201171875 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1939.71484375 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 245.15234375 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0•I$ž²‚(:“ êŠÓ‰ïÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.813' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.27099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04q"³‚(:¶  êŠÓÜ¢ƒ´ disk.ê@m¸drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1828.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 network.recv‚546730# disk.dm-0.out‚29368.40478515625 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.sda.out‚ 245.546875 disk.rbd1.in‚0 cpu‚25.31975293559109 disk.drbd1673.out‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚29368.40478515625 disk.nvme0n1.in‚ 83.52734375% disk./.usageGB‚1260.8535537719727 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1289.in‚0 network.sent‚ 9.330236e+06 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0( disk./.usagePercent‚70.5135312341618 disk.drbd2118.in‚0 disk.drbd1931.out‚0$ memory_percent‚92.10626284281412* proc.memory.percent‚11.829026540120442 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.rbd1.out‚ 48.2314453125 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1939.96484375 disk.sda2.out‚ 245.546875 disk.dm-0.in‚ 83.52734375 disk.drbd1510.out‚0Ò£öýœ´‚(:‘ ùŠÓº§Ñ³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.8' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.60000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E©€nFµ‚(:»  ùŠÓˆœò³ disk.drbd1817.out‚0& disk.nvme0n1.out‚29369.10400390625) proc.memory.availableMB‚ 1939.93359375 cpu‚25.319774977480275 network.recv‚546796 disk.drbd1584.in‚0 disk.drbd1931.out‚0# disk.dm-0.out‚29369.10400390625 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd2111.out‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 83.52734375" proc.memory.rssMB‚ 2907.1015625 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.sda.out‚ 245.93359375 disk.drbd2078.in‚0 disk.drbd1226.out‚0( disk./.usagePercent‚70.5135301674726 disk.drbd2118.out‚0 disk.rbd1.out‚ 48.2939453125% disk./.usageGB‚1260.8535346984863 disk.sda2.out‚ 245.93359375 disk.drbd2078.out‚0 disk.drbd1510.out‚0 network.sent‚ 9.330302e+06 disk.drbd1550.out‚0 disk.drbd1552.in‚0$ memory_percent‚92.10638999938965* proc.memory.percent‚11.829026540120442 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0Ä÷¨úž¶‚(:“ ˆÃŠÓ‡Óâ³ gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.433' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.81099999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ áA·‚(:¶  ˆÃŠÓ¼……´$ disk./.usageGB‚1260.853443145752 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 246.3515625 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 1939.9296875 disk.sda2.in‚0.03125 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd2041.in‚0 disk.drbd2111.out‚0$ memory_percent‚92.10640589396158 network.recv‚549650 disk.drbd1289.out‚0 disk.drbd1965.in‚0) disk./.usagePercent‚70.51352504736441 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 83.52734375# disk.dm-0.out‚29370.16162109375 disk.drbd1965.out‚0 disk.drbd1226.out‚0 network.sent‚ 9.33651e+06 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚29370.16162109375 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.829026540120442 disk.sda2.out‚ 246.3515625 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 48.4736328125 disk.drbd1285.out‚0 cpu‚25.31969092109328 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.rbd1.in‚0Ò,Ѓž¸‚(:“ —ÊÓŠ»¨³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.208' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.06933333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ê2¡F¹‚(:»  —ÊÓ؎ѳ disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8534622192383 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 83.52734375 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51352611405362 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0& disk.nvme0n1.out‚29370.45849609375 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 246.69140625 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚29370.45849609375 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 48.8095703125 disk.drbd2078.out‚0 network.recv‚549782 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.sda2.in‚0.03125 cpu‚25.31958414496054 disk.sda.in‚0.03125 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1939.93359375* proc.memory.percent‚11.829026540120442 disk.sda1.in‚0 network.sent‚ 9.337319e+06 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 246.69140625" proc.memory.rssMB‚ 2907.1015625 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.10638999938965Ûä“2º- _wandbruntime‚9560  _runtime‚9560êô ’Å»¿ _runtime‚ 9560.74881477 _step‚434 & lr-AdamW/pg1‚8.313790360576301e-05 & lr-AdamW/pg2‚8.313790360576301e-05 trainer/global_step‚5249 $ _timestamp‚1.784848797391551e+09›:6*Ê¼Ä & lr-AdamW/pg1‚8.313790360576301e-05 & lr-AdamW/pg2‚8.313790360576301e-05 trainer/global_step‚5249 $ _timestamp‚1.784848797391551e+09 _runtime‚ 9560.74881477 _step‚434² %ù2½- _wandbruntime‚9560  _runtime‚9560ƒðêÓ(¾ýO _runtime‚9560.753957088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5475479048704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48723194 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6696700297728 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94264128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5475479048704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15971914 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1509858100767120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45540940 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29960678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13988759 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15971532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13988764 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48733258 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413651607610368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 % _timestamp‚1.7848487973926709e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94274198 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45540934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48733258 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1515325781156935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1510454014113792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5466310350561 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1510454014113792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5476824484864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1515324411117681 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1515929493162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1510454040287232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29960291 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1510454040287232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚413651975701504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6696642739200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5467654961847 _step‚435 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45540940 I&(¿‚P M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94274198 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45540934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48733258 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1515325781156935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1510454014113792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5466310350561 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1510454014113792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5476824484864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1515324411117681 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1515929493162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1510454040287232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29960291 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1510454040287232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚413651975701504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6696642739200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5467654961847 _step‚435 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45540940 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94264128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5475479048704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15971914 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1509858100767120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45540940 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29960678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13988759 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15971532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13988764 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48733258 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413651607610368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 % _timestamp‚1.7848487973926709e+09³óߨ92À- _wandbruntime‚9562  _runtime‚9562¶ÅÝ£Á†N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6697889026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5468675763599 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94291661 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94281577 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1510136927968864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1510732949577728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94291661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚420428083002368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚5249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1510732923404288 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1516210796577792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1515605603732463 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5467331152201 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚ 9562.22585037 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45549307 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29965894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29966272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1510136902540896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reques”Ÿïàited_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1516210796577792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5477847000064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413730562067456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1516209424961024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚420428508644864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6697946577408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45549301 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13991399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48732276 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5476501556736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45549301 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48742354 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5477847000064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29965894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29966272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1510136902540896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1516210796577792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5477847000064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413730562067456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1516209424961024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚420428508644864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6697946577408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45549301 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13991399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48732276 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5476501556736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45549301 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48742354 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5477847000064 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1510732949577728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94291661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚420428083002368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 trainer/global_step‚5249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1510732923404288´‡¨1s2Ã- _wandbruntime‚9562  _runtime‚9562X yÄ 1loss/SRqqt1/detection_loss‚0.8721511700592036 trainer/global_step‚5249 0loss/FRt2/assignment_loss‚0.7659167323452327 2loss/SRqqt2/assignment_loss‚0.5278463202575571 1loss/FBt2/assignment_loss‚0.00805189273284955 /loss/FBt2/detection_loss‚0.5433414489787995 _step‚437 2loss/SRqqt1/assignment_loss‚0.4502686546630214 0loss/SRqqt2/detection_loss‚0.606612882172764 /loss/FBt1/detection_loss‚0.5380620209500193 epoch‚18 4loss/FBt1/assignment_loss‚0.00013012958162003467 /loss/FRt2/detection_loss‚1.4345346574146791 $loss/total_loss‚8.07089473591322 $ _timestamp‚1.784848798904702e+09 _runtime‚9562.261927881 /loss/FRt1/assignment_loss‚1.748865116458885 /loss/FRt1/detection_loss‚0.5751137102985875ˆ®ì ņ 4loss/FBt1/assignment_loss‚0.00013012958162003467 /loss/FRt2/detection_loss‚1.4345346574146791 $loss/total_loss‚8.07089473591322 $ _timestamp‚1.784848798904702e+09 _runtime‚9562.261927881 /loss/FRt1/assignment_loss‚1.748865116458885 /loss/FRt1/detection_loss‚0.5751137102985875 1loss/SRqqt1/detection_loss‚0.8721511700592036 trainer/global_step‚5249 0loss/FRt2/assignment_loss‚0.7659167323452327 2loss/SRqqt2/assignment_loss‚0.5278463202575571 1loss/FBt2/assignment_loss‚0.00805189273284955 /loss/FBt2/detection_loss‚0.5433414489787995 _step‚437 2loss/SRqqt1/assignment_loss‚0.4502686546630214 0loss/SRqqt2/detection_loss‚0.606612882172764 /loss/FBt1/detection_loss‚0.5380620209500193 epoch‚18µäد„GÆ‚(:¼  ¦ÃŠÓŒßÔ³ disk.rbd1.in‚0* proc.memory.percent‚11.829026540120442 disk.sda.in‚0.03125 disk.drbd1552.in‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8534812927246 disk.drbd1673.out‚0 disk.drbd1931.out‚0# disk.dm-0.out‚29371.33837890625 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1285.in‚0 disk.sda.out‚ 246.94921875 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 246.94921875 disk.drbd2041.out‚0$ memory_percent‚92.10654894510904 disk.drbd1552.out‚0& disk.nvme0n1.out‚29371.33837890625 disk.sda1.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 network.sent‚ 9.337427e+06) disk./.usagePercent‚70.51352718074281 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 cpu‚25.319501375313013 disk.drbd2118.out‚0) proc.memory.availableMB‚ 1939.89453125 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1817.out‚0 network.recv‚549890 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 48.8095703125 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0ÄE±“Ç‚(:ˆ ¦ÃŠÓ‘‡´³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚144.537' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚48.179 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 *1džÈ‚(:“ µÃŠÓœå¬³ gpu.0.gpu‚26.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.452' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.150666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02P ÙBÉ‚(:·  µÃŠÓùñѳ disk.drbd2111.in‚0 disk.rbd1.out‚ 48.8720703125 disk.sda.out‚ 247.328125 disk.sda2.out‚ 247.328125 disk.dm-0.in‚ 83.52734375 disk.drbd1356.in‚0 disk.drbd1828.out‚0 cpu‚25.31939305078093% disk./.usageGB‚1260.8534622192383 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125) disk./.usagePercent‚70.51352611405362 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1356.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1965.in‚0& disk.nvme0n1.out‚29371.52587890625 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1939.64453125 network.sent‚ 9.337493e+06 disk.rbd1.in‚0$ memory_percent‚92.10756619771323 disk.drbd1285.in‚0 network.recv‚549956# disk.dm-0.out‚29371.52587890625 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.sda2.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0féÝ“Ê‚(:ˆ ÄÊÓÀÂà³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚285.123' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.041 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«I•Ý8Ë‚(:­  ÄÊÓñÕý³ network.sent‚ 9.405256e+06 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 network.recv‚553301) disk./.usagePercent‚70.51352504736441 disk.drbd1931.in‚0 disk.drbd1931.out‚0# memory_percent‚92.1071688334147 disk.drbd1510.out‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda.out‚247.625 disk.drbd1226.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2078.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.rbd1.out‚ 48.9580078125 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0% disk.nvme0n1.out‚29371.8095703125" disk.dm-0.out‚29371.8095703125 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0( proc.memory.availableMB‚ 1939.7421875 disk.drbd1285.in‚0$ disk./.usageGB‚1260.853443145752 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 cpu‚25.319335039727555 disk.drbd2041.out‚0 disk.sda2.out‚247.625* proc.memory.percent‚11.829026540120442‘Xõ÷žÌ‚(:“ ÓÊÓز³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.155' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.71833333333332 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÙ}aCÍ‚(:¸  ÓÊÓ³êÓ³ disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1584.out‚0% disk.nvme0n1.out‚29371.9892578125 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1828.in‚0* proc.memory.percent‚11.829026540120442 network.recv‚553433 disk.nvme0n1.in‚ 83.52734375 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0# memory_percent‚92.1071688334147% disk./.usageGB‚1260.8534622192383 disk.drbd1289.in‚0 disk.drbd2111.out‚0 cpu‚25.319280859577116 disk.drbd2118.in‚0 disk.sda.out‚ 247.96484375 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 49.1220703125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51352611405362 disk.drbd1285.in‚0 disk.sda.in‚0.03125 network.sent‚ 9.406057e+06 disk.drbd1552.in‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1939.7421875 disk.drbd1673.in‚0 disk.sda2.out‚ 247.96484375" disk.dm-0.out‚29371.9892578125 disk.drbd1226.out‚0 disk.drbd1550.in‚0ÿ!q^žÎ‚(:“ âÊÓ¾ì̳ gpu.0.gpu‚25.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.854' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.284666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù“òAÏ‚(:¶  âÊÓÒð³ disk.drbd2041.out‚0 cpu‚25.31917209586866 disk.sda1.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 network.recv‚554380 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 network.sent‚ 9.411323e+06 disk.nvme0n1.in‚ 83.52734375 disk.sda2.out‚ 248.2890625 disk.drbd1289.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1817.out‚0# memory_percent‚92.1071688334147 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 49.1220703125 disk.drbd1931.in‚0" proc.memory.rssMB‚ 2907.1015625% disk./.usageGB‚1260.8532180786133& disk.nvme0n1.out‚29372.34912109375 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0( disk./.usagePercent‚70.5135124604318 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 248.2890625 disk.drbd1938.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0# disk.dm-0.out‚29372.34912109375( proc.memory.availableMB‚ 1939.7421875 disk.drbd1510.in‚0 disk.drbd1289.out‚0ŸW6EЂ(:º  ñÊÓÆðð³ disk.drbd1828.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚29372.54052734375 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 network.recv‚554446 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8532409667969 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0# disk.dm-0.out‚29372.54052734375 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 network.sent‚ 9.411389e+06 disk.drbd1965.out‚0 disk.drbd1510.out‚0# memory_percent‚92.1071688334147 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 49.1845703125 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 248.66015625 disk.dm-0.in‚ 83.52734375 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51351374045885 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1939.7421875* proc.memory.percent‚11.829026540120442 cpu‚25.319090698006104 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 248.66015625 disk.drbd2078.out‚0›«Ž¡žÑ‚(:“ ñÊÓö ¡´ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.64233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸúÒ2Ò- _wandbruntime‚9647  _runtime‚9647–ÍUEÇÓÁ & lr-AdamW/pg1‚8.252768385499221e-05 & lr-AdamW/pg2‚8.252768385499221e-05 trainer/global_step‚5299 % _timestamp‚1.7848488839891126e+09 _runtime‚9647.346311212 _step‚438‹9¿RÌÔÆ & lr-AdamW/pg1‚8.252768385499221e-05 & lr-AdamW/pg2‚8.252768385499221e-05 trainer/global_step‚5299 % _timestamp‚1.7848488839891126e+09 _runtime‚9647.346311212 _step‚438¶dÎÈ2Õ- _wandbruntime‚9647  _runtime‚9647Í5r©ˆ ÖýO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848488839901044e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _step‚439 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5517414233369 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45959290 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1529926147243520 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 trainer/global_step‚5299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1529927518853120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5526668604928 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1524399504812032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16120417 IDeviceStatsMonitor.on_train_batch_star(î€V{t/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1523798214883888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5528014041088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49189210 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1529315603689289 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1524399504812032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30241549 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1529926147243520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1529316973728543 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95138430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚9647.347632739 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5526668604928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30241176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424340259002880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1523798189455920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5528014041088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚417580925907968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49189210 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45959284 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45959290 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14120759 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49179146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1529926147243520 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 trainer/global_step‚5299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1529927518853120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5526668604928 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1524399504812032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16120417 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1523798214883888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5528014041088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49189210 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1529315603689289 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1524399504812032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30241549 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1529926147243520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1529316973728543 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95138430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚9647.347632739 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5526668604928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30241176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424340259002880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1523798189455920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5528014041088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚417580925907968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49189210 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45959284 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45959290 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14120759 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49179146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1530206079042048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5529036556288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45967657 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95165963 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1524077016657664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5519779646407 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49188228 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14123404 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1524076991229696 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30246744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45967651 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95155879 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5527691112960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 _step‚440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49188228 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1529596796304071 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5518435035009 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49198306 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424419666792448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45967657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16123345 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1524678387929088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1524678387929088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95155879 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚417659880365056 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1524678414102528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14123399 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848488854595513e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424420092434944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16123717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6760154518528 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5527691112960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚417659512273920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6760212069888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0,«¨lÚŒN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6760154518528 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5527691112960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚417659512273920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6760212069888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1530206079042048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5529036556288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45967657 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95165963 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1524077016657664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5519779646407 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_bataO8 ÷ch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49188228 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14123404 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1524076991229696 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30246744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45967651 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95155879 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5527691112960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 _step‚440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49188228 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1529596796304071 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5518435035009 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49198306 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424419666792448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45967657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16123345 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1524678387929088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1524678387929088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95155879 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚417659880365056 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1524678414102528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14123399 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848488854595513e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424420092434944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16123717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0¸" e2Û- _wandbruntime‚9648  _runtime‚9648ï2B| ܃ 4loss/FBt1/assignment_loss‚0.00020428773778100796 %loss/total_loss‚8.539362286109233 % _timestamp‚1.7848488855021574e+09 _runtime‚9648.859349233 _step‚441 0loss/FRt1/assignment_loss‚1.7918164906085738 /loss/FRt1/detection_loss‚0.5369820756862936 1loss/SRqqt1/detection_loss‚0.9310846328703674 2loss/SRqqt1/assignment_loss‚0.5661257804342906 2loss/SRqqt2/assignment_loss‚0.5394318841120105 /loss/FBt2/assignment_loss‚0.047753167461111 /loss/FRt2/detection_loss‚1.3954569433819881 1loss/SRqqt2/detection_loss‚0.6123515295515424 /loss/FBt1/detection_loss‚0.7321249442810024 /loss/FBt2/detection_loss‚0.7000362205527639 0loss/FRt2/assignment_loss‚0.6859943294315086 epoch‚18 trainer/global_step‚5299_äU݈ 4loss/FBt1/assignment_loss‚0.00020428773778100796 %loss/total_loss‚8.539362286109233 % _timestamp‚1.7848488855021574e+09 _runtime‚9648.859349233 _step‚441 0loss/FRt1/assignment_loss‚1.7918164906085738 /loss/FRt1/detection_loss‚0.5369820756862936 1loss/SRqqt1/detection_loss‚0.9310846328703674 2loss/SRqqt1/assignment_loss‚0.5661257804342906 2loss/SRqqt2/assignment_loss‚0.5394318841120105 /loss/FBt2/assignment_loss‚0.047753167461111 /loss/FRt2/detection_loss‚1.3954569433819881 1loss/SRqqt2/detection_loss‚0.6123515295515424 /loss/FBt1/detection_loss‚0.7321249442810024 /loss/FBt2/detection_loss‚0.7000362205527639 0loss/FRt2/assignment_loss‚0.6859943294315086 epoch‚18 trainer/global_step‚5299¹*µ ûŸÞ‚(:” €ÄŠÓ¶ÈÞ³ gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.303' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.43433333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÆ*Fß‚(:»  €ÄŠÓÊ쀴 disk.drbd1510.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1939.6796875 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1817.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 249.04296875 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 network.recv‚555370 disk.drbd1828.in‚0 network.sent‚ 9.416991e+06& disk.nvme0n1.out‚29372.90380859375 disk.drbd1552.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 49.1845703125 disk.drbd1673.out‚0 disk.drbd2041.out‚0% disk./.usageGB‚1260.8531455993652 disk.drbd1552.out‚0 disk.sda.out‚ 249.04296875 disk.drbd1931.in‚0 disk.drbd2078.out‚0" proc.memory.rssMB‚ 2907.1015625# disk.dm-0.out‚29372.90380859375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1356.in‚0 disk.drbd2111.out‚0 cpu‚25.319008919149574 disk.drbd2118.in‚0) disk./.usagePercent‚70.51350840701282 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1828.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2041.in‚0$ memory_percent‚92.10742314656576ýVàžà‚(:“ ÄŠÓÝÝͳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚291.361' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.12033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–EÂ9á‚(:®  ÄŠÓƒð³ proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0# disk.dm-0.out‚29373.07958984375 disk.drbd1510.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚249.25 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 49.404296875 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.829026540120442 network.sent‚ 9.477103e+06 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1550.in‚0 disk.sda.out‚249.25 disk.drbd1289.out‚0$ memory_percent‚92.10742314656576( proc.memory.availableMB‚ 1939.6796875 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 83.52734375& disk.nvme0n1.out‚29373.07958984375 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2907.1015625 network.recv‚558120 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 83.52734375) disk./.usagePercent‚70.51351160708043 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.8532028198242 disk.drbd1931.in‚0 disk.drbd2078.in‚0 cpu‚25.318928737515314?È;vžâ‚(:“ žÄŠÓâÔͳ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.21' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.40333333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¹QFã‚(:»  žÄŠÓæâî³) proc.memory.availableMB‚ 1940.08203125$ memory_percent‚92.10578600565592 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 83.52734375) disk./.usagePercent‚70.51351267376964 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.sda2.out‚ 249.58984375 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0# disk.dm-0.out‚29373.39013671875 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2907.1015625 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 network.recv‚559087 disk.drbd1673.in‚0 disk.drbd1938.in‚0& disk.nvme0n1.out‚29373.39013671875 disk.drbd1510.out‚0 network.sent‚ 9.478221e+06 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 49.404296875 disk.drbd1552.in‚0 disk.drbd2041.out‚0% disk./.usageGB‚1260.8532218933105 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1285.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.829026540120442 cpu‚25.320313673123124 disk.sda.in‚0.03125 disk.sda.out‚ 249.58984375 disk.sda2.in‚0.03125 disk.drbd1226.out‚0¹c†2ä- _wandbruntime‚9703  _runtime‚9703éŸ\;åÖv 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027778506278992 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9836889505386353 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5875740647315979 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 epoch‚18 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7191801071166992 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958508610725403 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6965921521186829 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4617958068847656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7375518679618835 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6134615540504456 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1971074789762497 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38358762860298157 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3849780559539795 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.750590980052948 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27305492758750916 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7721092104911804 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02344062551856041 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19870860874652863 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7467716336250305 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6917934417724609 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035241033881902695 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.779925525188446 /particle/accuracy_4_of_4‚0.9627353549003601 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.236111119389534 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20005127787590027 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4512820243835449 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.851749062538147 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2525249123573303 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954000115394592 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4528753161430359 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6331915259361267 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.651926577091217 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6234217286109924 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7461391091346741 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24194595217704773 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 % _timestamp‚1.7848489397655766e+09 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7805326581001282 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444626569748 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.382798433303833 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03183623403310776 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5867868661880493 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10995709151029587 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6693738102912903 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24194595217704773 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04914456605911255 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1255522221326828 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9983766674995422 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969591498374939 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8835963010787964 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5026353597640991 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06705495715141296 *jet/accuracy_2_of_4‚0.8480380177497864 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24164137244224548 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2002553939819336 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7367857694625854 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22433319687843323 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18632812798023224 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7698052525520325 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.830036997795105 *jet/accuracy_1_of_3‚0.9341450929641724 particle/accuracy_1_of_3‚1 _step‚442 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8814388513565063 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6894641518592834 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19850647449493408 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5746846199035645 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6244062185287476 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444626569748 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6526318192481995 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6494545936584473 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 *jet/accuracy_1_of_2‚0.7741455435752869 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3270266056060791 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8001082539558411 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7700102925300598 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚8.299682256407207 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9964285492897034 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8034943342208862 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21328957378864288 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.985741138458252 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.881944477558136 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21298468112945557 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977678656578064 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44725874066352844 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23382620513439178 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02344062551856041 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.801723837852478 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4473731815814972 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988095223903656 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2394515424966812 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869793057441711 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04228755831718445 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7753817439079285 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1846705973148346 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8302274346351624 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *jet/accuracy_1_of_4‚0.9788068532943726 (particle/accuracy‚0.8485513925552368 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027069849893450737 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5608441233634949 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29315730929374695 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.742393434047699 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0791935920715332 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11136205494403839 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.236111119389534 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19373014569282532 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7384615540504456 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6131271719932556 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_4‚0.09317515790462494 _runtime‚9703.123653541 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6654682755470276 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09600316733121872 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33415812253952026 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9837268590927124 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2554146945476532 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.688278317451477 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8024531006813049 trainer/global_step‚5319 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21261300146579742 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999404788017273 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.751226544380188 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2567780911922455 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2099677473306656 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8542845845222473 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_2‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.921345055103302 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7468532919883728 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02391738072037697 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2507326006889343 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6388940215110779 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7validation_average_jet_accuracy‚0.37344688177108765 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7024189233779907 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030875956639647484 /particle/accuracy_3_of_3‚0.9993678331375122 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.039329566061496735 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 +jet/accuracy_2_of_2‚0.15024355053901672 *particle/specificity‚0.909779965877533 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6669994592666626 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29957011342048645 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6554844975471497 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.651926577091217 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7001929879188538 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8835963010787964 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6744047999382019 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6417007446289062 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02771192416548729 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6863821148872375 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43329793214797974 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125NÍóÌ7æÛv 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19850647449493408 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5746846199035645 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6244062185287476 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444626569748 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6526318192481995 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6494545936584473 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 *jet/accuracy_1_of_2‚0.7741455435752869 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3270266056060791 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8001082539558411 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7700102925300598 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚8.299682256407207 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9964285492897034 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8034943342208862 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21328957378864288 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.985741138458252 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.881944477558136 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21298468112945557 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977678656578064 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44725874066352844 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23382620513439178 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02344062551856041 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.801723837852478 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4473731815814972 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988095223903656 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2394515424966812 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869793057441711 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04228755831718445 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7753817439079285 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1846705973148346 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8302274346351624 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *jet/accuracy_1_of_4‚0.9788068532943726 (particle/accuracy‚0.8485513925552368 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027069849893450737 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5608441233634949 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29315730929374695 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.742393434047699 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0791935920715332 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11136205494403839 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.236111119389534 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19373014569282532 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7384615540504456 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363746ælÚ**4735718 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6131271719932556 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_4‚0.09317515790462494 _runtime‚9703.123653541 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6654682755470276 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09600316733121872 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33415812253952026 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9837268590927124 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2554146945476532 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.688278317451477 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8024531006813049 trainer/global_step‚5319 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21261300146579742 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999404788017273 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.751226544380188 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2567780911922455 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2099677473306656 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8542845845222473 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_2‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.921345055103302 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7468532919883728 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02391738072037697 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2507326006889343 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6388940215110779 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7validation_average_jet_accuracy‚0.37344688177108765 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7024189233779907 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030875956639647484 /particle/accuracy_3_of_3‚0.9993678331375122 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.039329566061496735 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 +jet/accuracy_2_of_2‚0.15024355053901672 *particle/specificity‚0.909779965877533 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6669994592666626 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29957011342048645 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6554844975471497 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.651926577091217 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7001929879188538 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8835963010787964 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6744047999382019 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6417007446289062 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02771192416548729 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6863821148872375 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43329793214797974 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027778506278992 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9836889505386353 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5875740647315979 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 epoch‚18 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7191801071166992 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958508610725403 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6965921521186829 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4617958068847656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7375518679618835 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6134615540504456 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1971074789762497 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38358762860298157 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3849780559539795 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.750590980052948 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27305492758750916 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7721092104911804 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02344062551856041 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19870860874652863 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7467716336250305 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6917934417724609 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035241033881902695 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.779925525188446 /particle/accuracy_4_of_4‚0.9627353549003601 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.236111119389534 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20005127787590027 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4512820243835449 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.851749062538147 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2525249123573303 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954000115394592 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4528753161430359 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6331915259361267 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.651926577091217 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6234217286109924 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7461391091346741 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24194595217704773 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 % _timestamp‚1.7848489397655766e+09 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7805326581001282 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444626569748 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.382798433303833 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03183623403310776 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5867868661880493 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10995709151029587 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928570985794067 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6693738102912903 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24194595217704773 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04914456605911255 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1255522221326828 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9983766674995422 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969591498374939 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8835963010787964 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5026353597640991 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06705495715141296 *jet/accuracy_2_of_4‚0.8480380177497864 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24164137244224548 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2002553939819336 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7367857694625854 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22433319687843323 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18632812798023224 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7698052525520325 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.830036997795105 *jet/accuracy_1_of_3‚0.9341450929641724 particle/accuracy_1_of_3‚1 _step‚442 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8814388513565063 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6894641518592834º\9ïýžç‚(:“ ­ÄŠÓ°Óó gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚283.765' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.58833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lOªBEè‚(:º  ­ÄŠÓÑä³ disk.drbd1550.in‚0 network.sent‚ 9.48103e+06) disk./.usagePercent‚70.51362083605495 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 cpu‚25.323799743200905 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 1939.49609375 network.recv‚559219 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.sda2.out‚ 249.98828125 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 49.466796875 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8551559448242 disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚29374.11279296875 disk.sda1.out‚0$ memory_percent‚92.10817019144693* proc.memory.percent‚11.829026540120442 disk.dm-0.in‚ 83.52734375# disk.dm-0.out‚29374.11279296875 disk.nvme0n1.in‚ 83.52734375 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.sda.out‚ 249.98828125 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.sda1.in‚0ì„¥VAé‚(:¶  ¼ÄŠÓå›ó³ disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0$ disk.nvme0n1.out‚29374.384765625 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1828.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.829026540120442 network.recv‚559327 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 250.33203125 disk.drbd1673.in‚0% disk./.usageGB‚1260.8551559448242 disk.drbd1938.in‚0 disk.sda2.out‚ 250.33203125 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 cpu‚25.32373689736655 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2907.1015625! disk.dm-0.out‚29374.384765625 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1939.24609375 disk.drbd2078.out‚0 disk.drbd1356.out‚0 network.sent‚ 9.481138e+06 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 49.466796875 proc.cpu.threads‚17) disk./.usagePercent‚70.51362083605495 disk.drbd1285.out‚0$ memory_percent‚92.10918744405112'˜£$ê‚(:„ ¼ÄŠÓ÷‘Ò³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.6' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.2 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ww¾“ë‚(:ˆ ËÄŠÓæ‰â³ gpu.0.gpu‚47.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.541' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.847 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P€?ì‚(:´  ËĊӞу´ disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 network.sent‚ 9.540274e+06 disk.drbd1289.in‚0 disk.drbd1938.in‚0) proc.memory.availableMB‚ 1939.49609375" proc.memory.rssMB‚ 2907.1015625 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 network.recv‚562507 disk.drbd1828.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚29374.712890625 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0* proc.memory.percent‚11.829026540120442 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.rbd1.out‚ 49.6005859375 disk.drbd1931.in‚0 disk.drbd1356.in‚0 cpu‚25.323622967367644% disk./.usageGB‚1260.8551559448242 disk.drbd2041.in‚0$ memory_percent‚92.10817019144693 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.out‚0 disk.sda.out‚ 250.609375 disk.sda2.out‚ 250.609375 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 83.52734375 proc.cpu.threads‚17 disk.drbd1584.out‚0) disk./.usagePercent‚70.51362083605495! disk.dm-0.out‚29374.712890625 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0i5¾Zí‚(:’ ÚÄŠÓ˜‚ų gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚274.57' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.52333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­õ¤´=î‚(:²  ÚÄŠÓ¼þä³) disk./.usagePercent‚70.51362190274416 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.sda2.out‚250.9375 disk.drbd1356.in‚0 disk.rbd1.out‚ 49.6005859375 disk.drbd1510.in‚0" disk.dm-0.out‚29375.0400390625 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 network.recv‚563442 disk.sda.in‚0.03125 disk.dm-0.in‚ 83.52734375 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.nvme0n1.in‚ 83.52734375 disk.drbd1550.in‚0$ memory_percent‚92.10817019144693 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.8551750183105 disk.drbd1673.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1939.49609375" proc.memory.rssMB‚ 2907.1015625 disk.sda.out‚250.9375 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚29375.0400390625 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 cpu‚25.323536943177338 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1965.in‚0 network.sent‚ 9.541241e+06³ ¬2ï- _wandbruntime‚9753  _runtime‚9753´8ŠlÄð¾ % lr-AdamW/pg1‚8.19143230813517e-05 % lr-AdamW/pg2‚8.19143230813517e-05 trainer/global_step‚5349 % _timestamp‚1.7848489905771918e+09 _runtime‚ 9753.93442445 _step‚443ríløÉñà _runtime‚ 9753.93442445 _step‚443 % lr-AdamW/pg1‚8.19143230813517e-05 % lr-AdamW/pg2‚8.19143230813517e-05 trainer/global_step‚5349 % _timestamp‚1.7848489905771918e+09»L"_á2ò- _wandbruntime‚9753  _runtime‚9753¸|(óýO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96149471 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46467066 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚5349 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421559243718656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1546489096485888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16296987 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5584822033920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96159541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46467066 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5583476597760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46467072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1540297279295888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5575471566772 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1540905619888128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30556337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14259651 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1540905646061568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6828878858240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1540905619888128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16296686 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5574126955486 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46467072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5584822033920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421559611809792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1546490468095488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5583476597760 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1540905646061568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428388079698432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49682405 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1546489096485888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚9753.938958363 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49692469 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7848489905782464e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1546490468095488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428388490668032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49692469 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1545871406251374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 _step‚444™:™A ô‚P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1540905619888128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16296686 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5574126955486 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46467072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5584822033920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421559611809792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1546490468095488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5583476597760 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1540905646061568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428388079698432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49682405 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1546489096485888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚9753.938958363 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49692469 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7848489905782464e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 @:DeviceStatsMo½pînitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1546490468095488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428388490668032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49692469 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1545871406251374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 _step‚444 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96149471 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46467066 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 trainer/global_step‚5349 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421559243718656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1546489096485888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16296987 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5584822033920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96159541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46467066 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5583476597760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46467072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1540297279295888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5575471566772 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1540905619888128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30556337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14259651 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1540905646061568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6828878858240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864¼WÛu–2õ- _wandbruntime‚9755  _runtime‚9755Þu 'ö‡N SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _step‚445 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5584499105792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30561950 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5576492368524 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49701565 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428467960459776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 trainer/global_step‚5349 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _runtime‚9755.411246588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 408865280 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1546152598866156 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5585844549120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96166920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49701565 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96166920 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421638198175744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30562254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428468369325056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1546151228826790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1546769028284416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96177004 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6830130375168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96177004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1541184529178624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1540576081069664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1546770399901184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16299659 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1541184555352064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16299958 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1541184555352064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5585844549120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14262296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49691487 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49691487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1540576106497632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ρ'÷ŒN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96166920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49701565 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96166920 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421638198175744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30562254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428468369325056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1546151228826790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1546769028284416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96177004 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6830130375168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96177004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1541184529178624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1540576081069664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1546770399901184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16299659 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1541184555352064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16299958 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1541184555352064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5585844549120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14262296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49691487 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49691487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1540576106497632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _step‚445 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5584499105792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30561950 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5576492368524 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49701565 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428467960459776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 trainer/global_step‚5349 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _runtime‚9755.411246588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 408865280 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1546152598866156 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5585844549120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672½Õ/9è2ø- _wandbruntime‚9755  _runtime‚9755­4Õ$ù‚ /loss/FBt1/detection_loss‚0.7267848494793138 trainer/global_step‚5349 % _timestamp‚1.7848489920905712e+09 2loss/SRqqt1/assignment_loss‚0.6058881136776368 1loss/FBt1/assignment_loss‚0.04273114906012891 _runtime‚9755.447717159 %loss/total_loss‚8.956938793210043 0loss/FRt1/assignment_loss‚1.7890708549642653 0loss/FRt2/assignment_loss‚0.9392567672948727 /loss/FRt2/detection_loss‚1.4941051887540981 1loss/SRqqt2/detection_loss‚0.6572452469391922 _step‚446 1loss/SRqqt1/detection_loss‚0.8971975615048515 /loss/FBt2/detection_loss‚0.6675432981984046 epoch‚19 2loss/SRqqt2/assignment_loss‚0.5681517040872198 2loss/FBt2/assignment_loss‚0.028124692986759972 .loss/FRt1/detection_loss‚0.540839366263298‘Ñ ú‡ epoch‚19 2loss/SRqqt2/assignment_loss‚0.5681517040872198 2loss/FBt2/assignment_loss‚0.028124692986759972 .loss/FRt1/detection_loss‚0.540839366263298 /loss/FBt1/detection_loss‚0.7267848494793138 trainer/global_step‚5349 % _timestamp‚1.7848489920905712e+09 2loss/SRqqt1/assignment_loss‚0.6058881136776368 1loss/FBt1/assignment_loss‚0.04273114906012891 _runtime‚9755.447717159 %loss/total_loss‚8.956938793210043 0loss/FRt1/assignment_loss‚1.7890708549642653 0loss/FRt2/assignment_loss‚0.9392567672948727 /loss/FRt2/detection_loss‚1.4941051887540981 1loss/SRqqt2/detection_loss‚0.6572452469391922 _step‚446 1loss/SRqqt1/detection_loss‚0.8971975615048515 /loss/FBt2/detection_loss‚0.6675432981984046¾ep ;û‚(:°  éÄŠÓí÷س disk.dm-0.in‚ 83.52734375( disk./.usagePercent‚70.5136257428253 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.sda.in‚0.03125% disk./.usageGB‚1260.8552436828613 disk.sda2.in‚0.03125 disk.dm-0.out‚ 29375.265625 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0! disk.nvme0n1.out‚ 29375.265625 disk.sda1.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.rbd1.out‚ 49.6630859375 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 network.recv‚564471 disk.drbd1584.in‚0* proc.memory.percent‚11.829026540120442 disk.sda2.out‚ 251.13671875 disk.nvme0n1.in‚ 83.52734375 proc.cpu.threads‚17 disk.drbd1356.out‚0" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 9.546274e+06 disk.drbd1552.out‚0 disk.sda.out‚ 251.13671875 disk.drbd1931.in‚0 disk.drbd2118.in‚0( proc.memory.availableMB‚ 1939.4609375 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 cpu‚25.323450407226733 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1817.out‚0$ memory_percent‚92.10831324259439 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0`¼«¸œü‚(:‘ éÄŠÓÝ׸³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.2' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.06666666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•G(tBý‚(:·  øÄŠÓœÉγ disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 83.52734375$ memory_percent‚92.10831324259439 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0# disk.dm-0.out‚29375.72998046875 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 251.578125 disk.drbd1673.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.sda1.in‚0 network.recv‚566107 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 49.6630859375& disk.nvme0n1.out‚29375.72998046875" proc.memory.rssMB‚ 2907.1015625 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 83.52734375% disk./.usageGB‚1260.8552055358887 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 network.sent‚ 9.606236e+06 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.829026540120442 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 251.578125 disk.drbd2078.out‚0 disk.drbd1828.out‚0 cpu‚25.323389522039175( proc.memory.availableMB‚ 1939.4609375) disk./.usagePercent‚70.51362360944688 disk.drbd1356.out‚0}Š“ßþ‚(:“ øÄŠÓ“婳 gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚109.142' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.V†atŠ38066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚095 “ÿ‚(:ˆ ‡ÅŠÓ쇵³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚266.688' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03.@€‚(:µ  ‡ÅŠÓÑÕ³ disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2907.1015625% disk./.usageGB‚1260.8535270690918 disk.drbd1938.in‚0 cpu‚25.32327722818583 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1828.in‚0$ memory_percent‚92.10844039916992 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1356.in‚0 network.sent‚ 9.609225e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 49.8828125 disk.drbd1931.in‚0 disk.dm-0.in‚ 83.52734375 disk.rbd1.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 83.52734375* proc.memory.percent‚11.829026540120442 network.recv‚567099# disk.dm-0.out‚29375.94873046875 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0( proc.memory.availableMB‚ 1939.4296875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 251.9140625 disk.drbd2111.out‚0& disk.nvme0n1.out‚29375.94873046875) disk./.usagePercent‚70.51352974079691 disk.sda.out‚ 251.9140625 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd2118.out‚03ãKiž‚(:“ –ÅŠÓ¯ßÕ³ gpu.0.gpu‚66.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.629' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.87633333333335 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úïýÀ7‚‚(:¬  –ÅŠÓ¡ô³ disk.drbd1673.out‚0 disk.drbd1938.in‚0" proc.memory.rssMB‚ 2907.1015625) disk./.usagePercent‚70.51352974079691 disk.drbd1285.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 83.52734375 proc.cpu.threads‚17 network.sent‚ 9.612242e+06 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.10844039916992 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.sda.out‚252.25* proc.memory.percent‚11.829026540120442 disk.nvme0n1.in‚ 83.52734375 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0# disk.dm-0.out‚29376.36181640625 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1356.out‚0 network.recv‚568124 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚252.25 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8535270690918 disk.drbd1289.in‚0( proc.memory.availableMB‚ 1939.4296875& disk.nvme0n1.out‚29376.36181640625 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 49.8828125 disk.drbd1356.in‚0 cpu‚25.323189656231907 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0ÇR瓃‚(:ˆ ¥ÅŠÓŸ¥Ø³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚158.799' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚52.933 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶*Ö=„‚(:²  ¥ÅŠÓùˆý³ disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0$ memory_percent‚92.10844039916992% disk./.usageGB‚1260.8535842895508 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 252.7109375( proc.memory.availableMB‚ 1939.4296875 proc.cpu.threads‚17) disk./.usagePercent‚70.51353294086452 disk.drbd1931.in‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚29376.62744140625 disk.drbd1550.in‚0 disk.drbd1552.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.dm-0.in‚ 83.52734375 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0# disk.dm-0.out‚29376.62744140625 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 network.sent‚ 9.613217e+06 disk.drbd1285.out‚0 disk.rbd1.out‚50.125 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 cpu‚25.323128640251213 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 network.recv‚569011 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1550.out‚0 disk.sda.out‚ 252.7109375 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125«Ò&%“…‚(:ˆ ´ÅŠÓŸ×¾³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚251.385' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.795 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÐ/»7†‚(:¬  ´ÅŠÓ¿†Ý³ disk.drbd1828.in‚0 cpu‚25.323018894116128 network.recv‚569143 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚253.3125 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1939.42578125 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 83.52734375 disk.drbd2078.out‚0 disk.drbd1510.out‚0$ memory_percent‚92.10845629374185" proc.memory.rssMB‚ 2907.1015625 disk.sda2.in‚0.03125* proc.memory.percent‚11.829026540120442 disk.rbd1.out‚50.125 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8535079956055( disk./.usagePercent‚70.5135286741077 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚29377.86083984375 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.out‚253.3125 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 network.sent‚ 9.616014e+06& disk.nvme0n1.out‚29377.86083984375 disk.drbd1817.in‚0/¿?2‡- _wandbruntime‚9840  _runtime‚9840õM]LjÁ _runtime‚9840.837779307 _step‚447 & lr-AdamW/pg1‚8.129790168238615e-05 & lr-AdamW/pg2‚8.129790168238615e-05 trainer/global_step‚5399 % _timestamp‚1.7848490774805148e+09ÎûAÌ‰Æ & lr-AdamW/pg1‚8.129790168238615e-05 & lr-AdamW/pg2‚8.129790168238615e-05 trainer/global_step‚5399 % _timestamp‚1.7848490774805148e+09 _runtime‚9840.837779307 _step‚447¿­Ksg2Š- _wandbruntime‚9840  _runtime‚98400kBË(‹ýO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50148421 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432380988259328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6892058151936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1554237367984688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1560487122176512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97023773 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5636011590144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5634666153984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5625230838294 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1554851110586368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30836200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50138357 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425488562016256 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1559862598822982 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚5399 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1554237393412656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46885422 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1560487122176512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14391651 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46885416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97033843 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16444549 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5626575449580 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5636011590144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425488930107392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6891996399104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5634666153984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50138357 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1559863968862236 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560485750566912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30836559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1554851084412928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560485750566912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46885422 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50148421 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848490774816568e+09 _runtime‚9840.839141241 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0T²¹(Œ‚P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1560487122176512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14391651 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46885416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97033843 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16444549 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5626575449580 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5636011590144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425488930107392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6891996399104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5634666153984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50138357 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1559863968862236 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560485750566912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30836559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1554851084412928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560485750566912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46885422 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50148421 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848490774816568e+09 _runtime‚9840.839141241 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50148421 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432380988259328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6892058151936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1554237367984688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1560487122176512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97023773 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5636011590144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5634666153984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5625230838294 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1554851110586368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30836200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50138357 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425488562016256 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1559862598822982 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚5399 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1554237393412656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46885422 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248ÀöÙY2- _wandbruntime‚9842  _runtime‚9842.?¸'‰ Ž‡N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1555129993703424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7848490789609668e+09 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1560767053982208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16447424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6893244878336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97051306 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5637034105344 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1560143791437764 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30841715 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50157517 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14394291 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5637034105344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97051306 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425567516473344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6893302429696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚432460818903040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46893789 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1560142421398398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5627596251332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0#‰âp'ŒN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46893789 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1560142421398398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5627596251332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1555129993703424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7848490789609668e+09 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1560767053982208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16447424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6893244878336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97051306 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5637034105344 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1560143791437764 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30841715 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50157517 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14394291 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5637034105344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97051306 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425567516473344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6893302429696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚432460818903040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Á µ*¿2- _wandbruntime‚9842  _runtime‚9842ú¹’‘ 0loss/FRt2/assignment_loss‚0.6889631496247463 1loss/SRqqt1/assignment_loss‚0.613759695591947 2loss/SRqqt2/assignment_loss‚0.5468637005467397 /loss/FBt1/detection_loss‚0.8115188797028401 /loss/FBt2/detection_loss‚0.5427776014139117 %loss/total_loss‚8.528259721160275 trainer/global_step‚5399 _runtime‚9842.352283378 0loss/FRt1/assignment_loss‚1.7015243350019778 /loss/FRt2/detection_loss‚1.4598674520823187 1loss/FBt1/assignment_loss‚0.00412882178902553 2loss/FBt2/assignment_loss‚0.011093643979972546 /loss/FRt1/detection_loss‚0.5520976607894802 0loss/SRqqt2/detection_loss‚0.671906036811062 epoch‚19 % _timestamp‚1.7848490789950767e+09 1loss/SRqqt1/detection_loss‚0.9237587438262523 _step‚450uËá ’† %loss/total_loss‚8.528259721160275 trainer/global_step‚5399 _runtime‚9842.352283378 0loss/FRt1/assignment_loss‚1.7015243350019778 /loss/FRt2/detection_loss‚1.4598674520823187 1loss/FBt1/assignment_loss‚0.00412882178902553 2loss/FBt2/assignment_loss‚0.011093643979972546 /loss/FRt1/detection_loss‚0.5520976607894802 0loss/SRqqt2/detection_loss‚0.671906036811062 epoch‚19 % _timestamp‚1.7848490789950767e+09 1loss/SRqqt1/detection_loss‚0.9237587438262523 _step‚450 0loss/FRt2/assignment_loss‚0.6889631496247463 1loss/SRqqt1/assignment_loss‚0.613759695591947 2loss/SRqqt2/assignment_loss‚0.5468637005467397 /loss/FBt1/detection_loss‚0.8115188797028401 /loss/FBt2/detection_loss‚0.5427776014139117Âcnsž“‚(:“ ÃÅŠÓŽøس gpu.0.gpu‚39.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.229' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.40966666666665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ÑLÐ@”‚(:µ  ÃÅŠÓýíû³ disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1939.65234375 disk.drbd1965.out‚0* proc.memory.percent‚11.829026540120442 network.recv‚570712 disk.sda2.out‚ 253.59765625 disk.drbd1938.in‚0# disk.dm-0.out‚29378.09912109375 network.sent‚ 9.67815e+06 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1828.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 253.59765625 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8554496765137 disk.drbd2118.in‚0$ memory_percent‚92.10753440856934( disk./.usagePercent‚70.5136372630687& disk.nvme0n1.out‚29378.09912109375 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 83.52734375 disk.nvme0n1.in‚ 83.52734375 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.rbd1.out‚50.46875 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 cpu‚25.322957401091152 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0Ë›Išž•‚(:“ ÒÅŠÓìŠä³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.165' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.72166666666668 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯–«6–‚(:«  ÒÅŠÓẃ´( disk./.usagePercent‚70.5136372630687 disk.drbd1552.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚50.5625$ memory_percent‚92.10861523946127 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1939.38671875 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚254.1875 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1289.in‚0" disk.dm-0.out‚29379.2724609375 disk.drbd1584.in‚0 network.recv‚572544 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 network.sent‚ 9.681404e+06% disk./.usageGB‚1260.8554496765137 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.sda.out‚254.1875 disk.dm-0.in‚ 83.52734375 disk.nvme0n1.in‚ 83.52734375% disk.nvme0n1.out‚29379.2724609375 disk.drbd1584.out‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.829026540120442 cpu‚25.322897092079415Ö0|E/—‚(:¤  áÅŠÓÎ¥ì³ disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 cpu‚25.322813003265733 network.recv‚572676 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda.out‚254.5 disk.drbd1938.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1226.out‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2907.1015625 network.sent‚ 9.682208e+06 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125" disk.dm-0.out‚29379.6943359375 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 83.52734375 proc.cpu.threads‚17% disk.nvme0n1.out‚29379.6943359375 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0( disk./.usagePercent‚70.5136372630687 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1939.38671875% disk./.usageGB‚1260.8554496765137 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0$ memory_percent‚92.10861523946127 disk.sda2.out‚254.5 disk.drbd1356.out‚0 disk.rbd1.out‚50.625kÌ›“˜‚(:ˆ áÅŠÓîˆÏ³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚266.229' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.743 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…kŠü@™‚(:µ  ðÅŠÓþŽÚ³ disk.drbd2118.out‚0 disk.sda.out‚ 255.01171875 disk.drbd1938.out‚0 disk.drbd2118.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.sda2.in‚0.03125 disk.dm-0.in‚ 83.52734375 disk.drbd1673.in‚0 disk.sda2.out‚ 255.01171875 disk.drbd1552.in‚0 disk.sda1.in‚0$ memory_percent‚92.10861523946127 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0! disk.dm-0.out‚29380.244140625 disk.drbd1552.out‚0 disk.rbd1.out‚ 50.8046875 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1965.out‚0 cpu‚25.322754162824836 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8553733825684 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 network.sent‚ 9.687273e+06 disk.drbd1828.in‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1939.38671875 disk.drbd1289.out‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.829026540120442 network.recv‚573859) disk./.usagePercent‚70.51363299631188 disk.drbd1965.in‚0$ disk.nvme0n1.out‚29380.244140625 disk.drbd1226.out‚0 disk.drbd1356.out‚0FÌäpžš‚(:“ ðÅŠÓÓÙ·³ gpu.0.gpu‚40.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.438' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.47933333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠT± “›‚(:ˆ ÿÅŠÓÞÒ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.605' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.535 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JvmÎ9œ‚(:®  ÿÅŠÓ„¬ð³ disk.drbd1817.out‚0 disk.drbd1673.out‚0) proc.memory.availableMB‚ 1939.38671875 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.sda2.out‚ 255.3046875 disk.drbd1550.in‚0 disk.drbd2118.out‚0$ memory_percent‚92.10861523946127 disk.drbd2041.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1552.out‚0 network.sent‚ 9.687381e+06 network.recv‚573967 disk.drbd1285.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0! disk.dm-0.out‚29380.529296875 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51363747640654$ disk./.usageGB‚1260.855453491211 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125" proc.memory.rssMB‚ 2907.1015625* proc.memory.percent‚11.829026540120442 cpu‚25.3226686492761 disk.drbd1285.out‚0 disk.sda.out‚ 255.3046875 disk.drbd1938.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1356.in‚0 disk.drbd1356.out‚0$ disk.nvme0n1.out‚29380.529296875 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚50.96875 disk.sda1.out‚0¬É ,2- _wandbruntime‚9927  _runtime‚99275åÅž¿ # _timestamp‚1.78484916432112e+09 _runtime‚9927.678364098 _step‚451 & lr-AdamW/pg1‚8.067850045681808e-05 & lr-AdamW/pg2‚8.067850045681808e-05 trainer/global_step‚5449Ñ “RÊŸÄ & lr-AdamW/pg2‚8.067850045681808e-05 trainer/global_step‚5449 # _timestamp‚1.78484916432112e+09 _runtime‚9927.678364098 _step‚451 & lr-AdamW/pg1‚8.067850045681808e-05Ãè_ö 2 - _wandbruntime‚9927  _runtime‚9927[G.œˆ¡ýO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97898075 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1568796575111168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848491643225908e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1573855161433844 _step‚452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1568177482101456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚364 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97908145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚359 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16591842 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14523651 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50594309 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47303766 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31115134 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16591483 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚436372389550592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6954566795264 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1574482404647936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Wˆ[,¹{BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1574483776257536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6954509236736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1574483776257536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚429417880313856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5685855710208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1568796575111168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5676334721102 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50604373 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31115498 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚436372815200256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14523656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47303772 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1568796548937728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5677679332388 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5685855710208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47303766 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1568796548937728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1573853791394590 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1568177456673488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚9927.680096309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1574482404647936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5687201146368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50604373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5687201146368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚429418248404992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248˜;p(¢‚P WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5685855710208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1568796575111168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5676334721102 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50604373 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31115498 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚436372815200256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14523656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47303772 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1568796548937728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5677679332388 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5685855710208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47303766 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1568796548937728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1573853791394590 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1568177456673488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚9927.680096309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1574482404647936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5687201146368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50604373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5687201146368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚429418248404992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97898075 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1568796575111168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848491643225908e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1573855161433844 _step‚452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1568177482101456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚364 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97908145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚359 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16591842 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14523651 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50594309 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47303766 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31115134 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16591483 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚436372389550592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6954566795264 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1574482404647936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1574483776257536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6954509236736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1574483776257536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚429417880313856ÄüN/”2£- _wandbruntime‚9929  _runtime‚9929°ÎŠ '¤…N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31121015 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5686878218240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50613469 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚436452645010432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚453 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5686878218240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47312133 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5678700134140 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1574762336446464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5677355522742 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97915524 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5688223661568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97925608 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1569075458228224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1574134984009372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97925608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚429496466679808 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚9929.159276273 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47312139 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31120650 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47312133 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47312139 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1568456258447264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1574763708063232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955752688128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50603391 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1568456283875232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1569075484401664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚436452645010432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚453 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5686878218240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47312133 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5678700134140 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1574762336446464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5677355522742 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97915524 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5688223661568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97925608 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1569075458228224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1574134984009372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97925608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚429496466679808 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚9929.159276273 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47312139 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31120650 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47312133 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47312139 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1568456258447264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1574763708063232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955752688128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50603391 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 _JDeviceStatsMonitor.on_train_batch_enS¨™ód/requested_bytes.large_pool.allocated‚1568456283875232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1569075484401664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5686878218240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50613469 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14655656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1588479058728960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1588479058728960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16738986 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51060325 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47722116 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1582742039635968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5737045266432 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1587844983966198 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51050261 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1582742039635968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚5499 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440364048109568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1582742013462528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47722116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚363 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1582117545362288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16738623 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5728783215196 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5727438603910 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5738390702592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚433347566702592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5738390702592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47722122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51060325 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51050261 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31394642 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7016904959488 _runtime‚10014.732917954 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98772377 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1588480430338560ÎÑÿ뺂P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14655651 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5737045266432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14655656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1588479058728960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1588479058728960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16738986 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51060325 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47722116 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1582742039635968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5737045266432 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1587844983966198 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51050261 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1582742039635968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ¸" ¿òtrainer/global_step‚5499 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440364048109568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1582742013462528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47722116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚363 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1582117545362288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16738623 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5728783215196 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5727438603910 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5738390702592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚433347566702592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5738390702592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47722122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51060325 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51050261 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31394642 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7016904959488 _runtime‚10014.732917954 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98772377 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1588480430338560 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1582117570790256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1588480430338560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31394274 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚457 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1583020922753024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98799910 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51059343 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31399849 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7018149862912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5729804016948 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47730483 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1588124806541614 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10016.20735626 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98799910 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5739413217792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51059343 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1588760362144256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848492528497527e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14658291 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440443877288960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1588758990527488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7018092311552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14658296 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47730483 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1583020922753024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1588126176580980•0­'½ŒN QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16741558 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1583020948926464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47730489 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5739413217792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5739413217792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51059343 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1588760362144256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848492528497527e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14658291 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440443877288960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1588758990527488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7018092311552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14658296 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47730483 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1583020922753024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1588126176580980 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16741916 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1582396347136064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚358 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98789826 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚363 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚457 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1583020922753024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98799910 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51059343 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31399849 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7018149862912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5729804016948 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47730483 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1588124806541614 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10016.20735626 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98799910 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48140472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444355762434560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437276885000192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1602475712809984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 _step‚460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99646679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51516277 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1596687504160768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7079292598272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5789580258816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1602477084419584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5788234822656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51516277 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1601836176537806 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1602475712809984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51506213 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5779887098004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16886655 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1596057634051088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10101.753002203 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1596057659479056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7848493383955326e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48140472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787651 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31673938 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48140466 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1596687477987328 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99656749 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1596687477987328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1602477084419584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31674311 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16886287 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51506213 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5778542486718#?9 (ÒƒP % _timestamp‚1.7848493383955326e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48140472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787651 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31673938 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48140466 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1596687477987328 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99656749 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1596687477987328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1602477084419584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31674311 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16886287 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51506213 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5778542486718 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14787656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48140466 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1596687504160768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5789580258816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7079245525504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5788234822656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚444356177598464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48140472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444355762434560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437276885000192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1602475712809984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 _step‚460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99646679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51516277 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1596687504160768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7079292598272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5789580258816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1602477084419584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5788234822656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51516277 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1601836176537806 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1602475712809984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51506213 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5779887098004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16886655 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1596057634051088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10101.753002203 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1596057659479056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136ÌmyàÁ4Ó/ _wandbruntime‚10103  _runtime‚10103cª·†HÔˆN LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1602757016225280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5789257330688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5790602774016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437355471366144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848493398728406e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1602755644608512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7080540644352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99664128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596966387277824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1596336435824864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99674212 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48148839 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99674212 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1596966387277824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51515295 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14790296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inacŽñÆtive_split.large_pool.current‚5 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31679489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14790291 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48148833 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444435588465152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16889562 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1602117369152588 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444436012010496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚5549 _runtime‚10103.230543535 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5780907899756 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437355103275008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51525373 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1596336461252832 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51515295 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5790602774016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚100782®'ÕN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437355103275008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51525373 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1596336461252832 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51515295 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5790602774016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1602757016225280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5789257330688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5790602774016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437355471366144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848493398728406e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1602755644608512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7080540644352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99664128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596966387277824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1596336435824864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99674212 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48148839 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99674212 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1596966387277824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51515295 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14790296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31679489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14790291 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48148833 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444435588465152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16889562 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1602117369152588 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444436012010496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚5549 _runtime‚10103.230543535 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5780907899756 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56Í0HÇ•4Ö/ _wandbruntime‚10103  _runtime‚10103¥Š ×… /loss/FRt1/detection_loss‚0.5645786518853253 1loss/SRqqt2/detection_loss‚0.6330720454960492 trainer/global_step‚5549 _runtime‚10103.266489563 _step‚462 2loss/SRqqt1/assignment_loss‚0.5557649748160528 2loss/SRqqt2/assignment_loss‚0.4863564146470415 3loss/FBt1/assignment_loss‚0.0025547844130670177 epoch‚19 0loss/FBt2/detection_loss‚0.44211075267736727 %loss/total_loss‚8.518049561449386 0loss/FRt1/assignment_loss‚1.7146164840023692 0loss/FRt2/assignment_loss‚0.6992574872375911 2loss/FBt2/assignment_loss‚0.025256474972473356 /loss/FRt2/detection_loss‚1.4803580580887925 1loss/SRqqt1/detection_loss‚1.0379098777894484 .loss/FBt1/detection_loss‚0.876213555423809 $ _timestamp‚1.784849339909295e+09›G‚‹ØŠ 0loss/FBt2/detection_loss‚0.44211075267736727 %loss/total_loss‚8.518049561449386 0loss/FRt1/assignment_loss‚1.7146164840023692 0loss/FRt2/assignment_loss‚0.6992574872375911 2loss/FBt2/assignment_loss‚0.025256474972473356 /loss/FRt2/detection_loss‚1.4803580580887925 1loss/SRqqt1/detection_loss‚1.0379098777894484 .loss/FBt1/detection_loss‚0.876213555423809 $ _timestamp‚1.784849339909295e+09 /loss/FRt1/detection_loss‚0.5645786518853253 1loss/SRqqt2/detection_loss‚0.6330720454960492 trainer/global_step‚5549 _runtime‚10103.266489563 _step‚462 2loss/SRqqt1/assignment_loss‚0.5557649748160528 2loss/SRqqt2/assignment_loss‚0.4863564146470415 3loss/FBt1/assignment_loss‚0.0025547844130670177 epoch‚19Îã¼â.‘Ù‚(:† ÂÇŠÓ¤—½³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.89' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.63 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*¾…CÚ‚(:¸  ÂÇŠÓëçܳ disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚29384.5732421875 disk.drbd2078.out‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51379961316559 disk.drbd1584.in‚0 disk.drbd1965.out‚0( proc.memory.availableMB‚ 1939.7890625 cpu‚25.321642170590287 disk.drbd1285.out‚0 network.recv‚590940 disk.drbd1817.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 network.sent‚ 9.936783e+06 disk.sda2.out‚ 259.66015625 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 51.673828125* proc.memory.percent‚11.829026540120442 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.sda.in‚0.03125 disk.drbd2041.out‚0$ memory_percent‚92.10697809855142 disk.sda.out‚ 259.66015625 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0" proc.memory.rssMB‚ 2907.1015625% disk./.usageGB‚1260.8583526611328 disk.drbd2111.out‚0% disk.nvme0n1.out‚29384.5732421875)µ´(žÛ‚(:“ ÑÇŠÓ‚¸½³ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.747' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.91566666666667 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÉhÐ@Ü‚(:µ  ÑÇŠÓ¼½Þ³% disk./.usageGB‚1260.8583335876465 disk.drbd1510.in‚0* proc.memory.percent‚11.829026540120442 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0" disk.dm-0.out‚29384.9755859375 disk.drbd1584.in‚0 disk.sda2.out‚ 260.2109375 disk.drbd1931.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 51.916015625% disk.nvme0n1.out‚29384.9755859375 disk.drbd1289.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 network.recv‚591890 disk.drbd1552.out‚0 disk.sda.out‚ 260.2109375 disk.drbd1673.in‚0 disk.drbd1965.in‚0" proc.memory.rssMB‚ 2907.1015625 cpu‚25.32158554819342 network.sent‚ 9.939758e+06 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1550.in‚0( proc.memory.availableMB‚ 1940.0390625 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 83.52734375) disk./.usagePercent‚70.51379854647638$ memory_percent‚92.10596084594727§F‘Ý‚(:† àÇŠÓ¨Ó³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚134.94' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚44.98 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À‘6Þ‚(:«  àÇŠÓ¯Ñó³ disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1550.out‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 cpu‚25.32147912224149 disk.drbd1356.out‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.829026540120442 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 83.52734375 disk.sda.in‚0.03125) disk./.usagePercent‚70.51379854647638 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 51.916015625 disk.sda.out‚ 260.53125 network.sent‚ 9.939824e+06 disk.drbd1356.in‚0 disk.drbd2041.out‚0$ memory_percent‚92.10697809855142 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.dm-0.out‚ 29385.3671875" disk.nvme0n1.out‚ 29385.3671875 disk.drbd2118.in‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8583335876465 disk.drbd1285.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1828.out‚0( proc.memory.availableMB‚ 1939.7890625 disk.sda2.out‚ 260.53125 disk.drbd1289.in‚0 disk.drbd1938.in‚0 network.recv‚591956 disk.drbd2111.in‚0 ç+$“ß‚(:ˆ ïÇŠÓñêß³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚267.354' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dóå@@à‚(:µ  ïÇŠÓÓˆ´ disk.drbd1510.in‚0 disk.nvme0n1.in‚ 83.52734375( proc.memory.availableMB‚ 1939.7890625 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0$ memory_percent‚92.10697809855142 cpu‚25.32142243855446 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 260.91796875! disk.dm-0.out‚29386.162109375 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚29386.162109375) disk./.usagePercent‚70.51370233111017 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 52.080078125" proc.memory.rssMB‚ 2907.1015625 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1552.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 260.91796875 disk.drbd1931.out‚0* proc.memory.percent‚11.829026540120442 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 network.sent‚ 9.945381e+06 network.recv‚592902 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8566131591797 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0ëAÓŸá‚(:” þÇŠÓáˆÂ³ gpu.0.gpu‚43.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.441' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.14699999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£¦“@â‚(:µ  þÇŠÓØ“á³ disk.sda.in‚0.03125$ memory_percent‚92.10596084594727 disk.sda2.out‚ 261.109375 disk.drbd2041.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51370382447504 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚29386.53076171875 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 261.109375 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17* proc.memory.percent‚11.829026540120442 disk.dm-0.in‚ 83.52734375 disk.rbd1.out‚ 52.345703125 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.out‚0 network.recv‚594710 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 cpu‚25.32134321511839 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0( proc.memory.availableMB‚ 1940.0390625" proc.memory.rssMB‚ 2907.1015625 disk.drbd1828.in‚0 network.sent‚ 9.948635e+06 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚29386.53076171875% disk./.usageGB‚1260.8566398620605 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0.%áÊ“ã‚(:ˆ ÈŠÓژೠgpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.519' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.173 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R×ý–Eä‚(:º  ÈŠÓè¼ù³ disk.nvme0n1.in‚ 83.52734375 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 cpu‚25.321261970498014 disk.drbd1550.in‚0) disk./.usagePercent‚70.51370382447504 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 network.recv‚595570 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8566398620605 disk.drbd1356.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 261.51953125 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 52.423828125& disk.nvme0n1.out‚29387.19970703125 disk.sda.in‚0.03125 disk.drbd2041.in‚0$ memory_percent‚92.10799535115561* proc.memory.percent‚11.829026540120442 disk.drbd1931.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1965.in‚0 network.sent‚ 9.951622e+06 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.sda.out‚ 261.51953125 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0# disk.dm-0.out‚29387.19970703125 disk.drbd1828.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2907.1015625 disk.drbd1938.out‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 1939.5390625 disk.sda1.out‚0 disk.drbd1226.in‚0ܦM–4å/ _wandbruntime‚10188  _runtime‚10188×thÈæ % _timestamp‚1.7848494256169906e+09 _runtime‚10188.974267365 _step‚463 & lr-AdamW/pg1‚7.880323160262611e-05 & lr-AdamW/pg2‚7.880323160262611e-05 trainer/global_step‚5599þ‚¬ÙÍçÇ % _timestamp‚1.7848494256169906e+09 _runtime‚10188.974267365 _step‚463 & lr-AdamW/pg1‚7.880323160262611e-05 & lr-AdamW/pg2‚7.880323160262611e-05 trainer/global_step‚5599Ï\(%4è/ _wandbruntime‚10188  _runtime‚10188­é·|> é‚P PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51972229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448347912966144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441205835206656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48558816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1610632968685568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5839424378880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1616472366891008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5829646369526 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51962165 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1610632942512128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7141647915520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48558822 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441206203297792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51962165 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48558816 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1610632968685568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 trainer/global_step‚5599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5830990980812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5839424378880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448347483122176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51972229 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7141709668352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1609997722739888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚387 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17035108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚10188.975677924 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1616472366891008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100520981 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48558822 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5840769815040 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1610632942512128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100531051 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6•ÕÍË (ê‡P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17035108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚10188.975677924 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1616472366891008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100520981 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48558822 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5840769815040 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1610632942512128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100531051 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51972229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448347912966144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441205835206656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48558816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1610632968685568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5839424378880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1616472366891008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5829646369526 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51962165 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1610632942512128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7141647915520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48558822 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441206203297792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51962165 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48558816 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1610632968685568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 trainer/global_step‚5599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5830990980812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5839424378880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448347483122176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51972229 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7141709668352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1609997722739888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚387 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181ÐmÒ…4ë/ _wandbruntime‚10190  _runtime‚10190$«ñY'ìŒN [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1610911851802624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5840446886912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48567189 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1610911877976064 _runtime‚10190.457416997 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31959921 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100548514 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1616753670306304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441284421572608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51971247 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1616752298689536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100538430 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1610911877976064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832011782564 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1610911851802624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14922291 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1616753670306304 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48567189 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51971247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51981325 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5840446886912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848494270997953e+09 _step‚465 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1610276524513664 trainer/global_step‚5599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441284789663744 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17037630 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5841792330240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14922296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48567183 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5830667171166 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚385 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448427322489344 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1616107191684830 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5841792330240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51981325 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7142900916736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448427748131840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100548514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17038015 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448427322489344 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1616107191684830 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5841792330240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51981325 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7142900916736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448427748131840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100548514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17038015 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1610911877976064 _runtime‚10190.457416997 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31959921 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100548514 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1616753670306304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441284421572608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51971247 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1616752298689536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100538430 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1610911877976064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832011782564 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1610911851802624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14922291 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1616753670306304 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48567189 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51971247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51981325 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5840446886912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848494270997953e+09 _step‚465 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1610276524513664 trainer/global_step‚5599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181Ñ0ã'Ñ4î/ _wandbruntime‚10190  _runtime‚10190ß"Ô ï… 3loss/FBt1/assignment_loss‚0.0010390318546700745 1loss/FBt2/assignment_loss‚0.04511175168122345 /loss/FBt2/detection_loss‚0.4938437953355171 % _timestamp‚1.7848494271326406e+09 _runtime‚10190.489818716 epoch‚19 0loss/FRt2/assignment_loss‚0.6420204135504636 2loss/SRqqt1/assignment_loss‚0.5160284743413746 /loss/FRt2/detection_loss‚1.3749499006080441 1loss/SRqqt2/detection_loss‚0.6499508908167017 1loss/SRqqt1/detection_loss‚0.9425971832795871 _step‚466 %loss/total_loss‚8.291947492450241 0loss/FRt1/assignment_loss‚1.7591562819315492 2loss/SRqqt2/assignment_loss‚0.4938307331221987 /loss/FRt1/detection_loss‚0.5485050999861281 /loss/FBt1/detection_loss‚0.8249139359427823 trainer/global_step‚5599»@a»ðŠ 3loss/FBt1/assignment_loss‚0.0010390318546700745 1loss/FBt2/assignment_loss‚0.04511175168122345 /loss/FBt2/detection_loss‚0.4938437953355171 % _timestamp‚1.7848494271326406e+09 _runtime‚10190.489818716 epoch‚19 0loss/FRt2/assignment_loss‚0.6420204135504636 2loss/SRqqt1/assignment_loss‚0.5160284743413746 /loss/FRt2/detection_loss‚1.3749499006080441 1loss/SRqqt2/detection_loss‚0.6499508908167017 1loss/SRqqt1/detection_loss‚0.9425971832795871 _step‚466 %loss/total_loss‚8.291947492450241 0loss/FRt1/assignment_loss‚1.7591562819315492 2loss/SRqqt2/assignment_loss‚0.4938307331221987 /loss/FRt1/detection_loss‚0.5485050999861281 /loss/FBt1/detection_loss‚0.8249139359427823 trainer/global_step‚5599ÒÐ…I‚Ÿñ‚(:” œÈŠÓªýس gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚118.771' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.590333333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¾?ò‚(:´  œÈŠÓáÍú³ disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8585243225098 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.829058329264324 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 52.423828125 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1965.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 83.52734375 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 network.recv‚595744) disk./.usagePercent‚70.51380921336843 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1939.484375 disk.sda2.out‚ 261.859375# disk.dm-0.out‚29387.44970703125 disk.drbd1226.in‚0 disk.drbd1673.in‚0! proc.memory.rssMB‚ 2907.109375 cpu‚25.323502679684786 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚29387.44970703125 network.sent‚ 9.952461e+06 disk.drbd1510.out‚0 disk.sda.out‚ 261.859375 disk.drbd1289.out‚0 disk.dm-0.in‚ 83.52734375$ memory_percent‚92.10821787516275 disk.drbd1673.out‚0 disk.drbd1931.in‚0êH¿4ó/ _wandbruntime‚10209  _runtime‚10209XÔÓ{­:ô§u 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6096999049186707 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6981752514839172 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.645875096321106 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7749160528182983 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 (jet/accuracy_1_of_2‚0.77460777759552 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6443178057670593 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6422106027603149 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30642205476760864 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26147687435150146 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 (jet/accuracy_1_of_1‚0.34400674700737 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8001082539558411 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6001492738723755 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22289077937602997 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030966728925704956 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044492367655038834 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.15276266634464264 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7356997132301331 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9257652163505554 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997065007686615 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23627527058124542 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02934188023209572 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21560774743556976 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )particle/specificity‚0.90782630443573 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6726378202438354 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024217380210757256 )loss/val_total_loss‚8.306270076755483 6validation_average_jet_accuracy‚0.3809753954410553 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.801723837852478 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02363685891032219 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08298216015100479 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6525034308433533 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28189411759376526 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8004689812660217 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43059206008911133 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02363685891032219 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38169360160827637 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7839517593383789 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6778760552406311 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4445217251777649 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03433651104569435 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7053388953208923 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4410119652748108 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19243808090686798 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4358656108379364 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46658483147621155 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2943895757198334 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6260010600090027 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763388991355896 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6368131637573242 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20624004304409027 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03662508726119995 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04121128097176552 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_1_of_4‚0.9707711338996887 (particle/accuracy‚0.8474469780921936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5858071446418762 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2610592544078827 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _step‚467 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3404218256473541 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7737491726875305 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26996535062789917 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12910331785678864 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.237111434340477 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7848494464751744e+09 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7978935837745667 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9921409487724304 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22978876531124115 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6892520785331726 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7107964754104614 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8564038276672363 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2844003736972809 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14215807616710663 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6761512160301208 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22496424615383148 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2235395610332489 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7692307829856873 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6531528830528259 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7791218757629395 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2065819799900055 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5365538001060486 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 *jet/accuracy_2_of_3‚0.5716387629508972 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12450077384710312 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7212464213371277 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7163178324699402 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.645875096321106 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27072200179100037 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11187051236629486 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 *jet/accuracy_2_of_4‚0.8464924693107605 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978915452957153 +jet/accuracy_3_of_3‚0.06796097010374069 epoch‚19 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6920003890991211 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8632078170776367 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5912014245986938 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38145509362220764 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6568951606750488 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_3_of_3‚1 .particle/accuracy_4_of_4‚0.953549861907959 trainer/global_step‚5599 _runtime‚10209.83328071 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6481091380119324 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7346394658088684 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8984130620956421 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7365312576293945 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7403279542922974 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7355321049690247 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4039163887500763 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.722184956073761 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04953497275710106 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18942680954933167 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5384615659713745 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762664794921875 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6379557847976685 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2974359095096588 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.801510214805603 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7971901893615723 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6316118240356445 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.10990224778652191 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3258592188358307 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27072200179100037 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24045529961585999 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7402691841125488 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 +particle/sensitivity‚0.6264205574989319 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6378995180130005 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6699026226997375 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2834865152835846 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2264915555715561 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762664794921875 )jet/accuracy_1_of_3‚0.923482358455658 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890350103378296 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0377555675804615 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_3_of_4‚1®³@\$õ¬u :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46658483147621155 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2943895757198334 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6260010600090027 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763388991355896 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6368131637573242 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20624004304409027 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03662508726119995 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04121128097176552 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_1_of_4‚0.9707711338996887 (particle/accuracy‚0.8474469780921936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5858071446418762 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2610592544078827 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _step‚467 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3404218256473541 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7737491726875305 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26996535062789917 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12910331785678864 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.237111434340477 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7848494464751744e+09 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7978935837745667 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9921409487724304 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22978876531124115 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6892520785331726 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7107964754104614 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8564038276672363 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2844003736972809 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14215807616710663 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6761512160301208 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22496424615383148 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2235395610332489 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7692307829856873 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6531528830528259 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7791218757629395 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2065819799900055 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5365538001060486 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 *jet/accuracy_2_of_3‚0.5716387629508972 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12450077384710312 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7212464213371277 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7163178324699402 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.645875096321106 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27072200179100037 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11187051236629486 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 *jet/accuracy_2_of_4‚0.8464924693107605 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978915452957153 +jet/accuracy_3_of_3‚0.06796097010374069 epoch‚19 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6920003890991211 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8632078170776367 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5912014245986938 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38145509362220764 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6568951606750488 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_3_of_3‚1 .particle/accuracy_4_of_4‚0.953549861907959 trainer/global_step‚5599 _runtime‚10209.83328071 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6481091380119324 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7346394658088684 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8984130620956421 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7365312576293945 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7403279542922974 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7355321049690247 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4039163887500763 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.722184956073761 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04953497275710106 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18942680954933167 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5384615659713745 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762664794921875 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6379557847976685 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2974359095096588 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.801510214805603 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7971901893615723 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6316118240356445 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.10990224778652191 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3258592188358307 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27072200179100037 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24045529961585999 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7402691841125488 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 +particle/sensitivity‚0.6264205574989319 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6378995180130005 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6699026226997375 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2834865152835846 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2264915555715561 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762664794921875 )jet/accuracy_1_of_3‚0.923482358455658 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890350103378296 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0377555675804615 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6096999049186707 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6981752514839172 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.645875096321106 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7749160528182983 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/1FRt2SRqqt1FBt/SRqqt_ò™òVpurity‚0.4583333134651184 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 (jet/accuracy_1_of_2‚0.77460777759552 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6443178057670593 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6422106027603149 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30642205476760864 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26147687435150146 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 (jet/accuracy_1_of_1‚0.34400674700737 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8001082539558411 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6001492738723755 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22289077937602997 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030966728925704956 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044492367655038834 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.15276266634464264 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7356997132301331 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9257652163505554 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997065007686615 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23627527058124542 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02934188023209572 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21560774743556976 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )particle/specificity‚0.90782630443573 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6726378202438354 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024217380210757256 )loss/val_total_loss‚8.306270076755483 6validation_average_jet_accuracy‚0.3809753954410553 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.801723837852478 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02363685891032219 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08298216015100479 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6525034308433533 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28189411759376526 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8004689812660217 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43059206008911133 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02363685891032219 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38169360160827637 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7839517593383789 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6778760552406311 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4445217251777649 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03433651104569435 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7053388953208923 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4410119652748108 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19243808090686798 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4358656108379364ÓU¼V²‘ö‚(:† «ÈŠÓ¶©æ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚57.0 gpu.0.powerWatts‚105.333' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.111 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü-œêF÷‚(:»  «ÈŠÓ敃´ disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0* proc.memory.percent‚15.774106979370117 disk.sda1.in‚0 disk.sda2.out‚ 262.21484375 disk.drbd1356.in‚0 disk.drbd1510.in‚0% disk./.usageGB‚1261.8622817993164 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.sda.out‚ 262.21484375 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚29979.75048828125 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚29989.63720703125( proc.memory.availableMB‚ 976.09765625 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.56994458609319 disk.drbd1584.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 cpu‚25.324858394564586 disk.sda.in‚0.03125 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 83.52734375# proc.memory.rssMB‚ 3876.64453125 network.sent‚ 9.952527e+06 disk.drbd1938.out‚0 network.recv‚595810 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0$ memory_percent‚96.02824846903482 disk.drbd2118.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1965.out‚0 disk.rbd1.out‚ 52.783203125T‘MnCø‚(:¸  ºÈŠÓ€º´ disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚31343.03564453125 network.recv‚598588 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 52.783203125 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51381156008468 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 262.6484375 disk.drbd1510.in‚0$ memory_percent‚88.12948862711588 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0% disk./.usageGB‚1260.8585662841797 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.rbd1.in‚0 cpu‚25.324383682769263 disk.drbd1931.in‚0" proc.memory.rssMB‚ 2907.1328125 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0' proc.memory.availableMB‚ 2917.296875 disk.drbd2118.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚31343.03564453125 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.sda.in‚0.03125 disk.drbd1938.out‚0 network.sent‚ 1.0032427e+07 disk.sda.out‚ 262.6484375 disk.drbd2041.out‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.829153696695965 proc.cpu.threads‚17:«ümžù‚(:“ ºÈŠÓç›â³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚302.32' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.77333333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gâžú‚(:“ ÉȊӧೠgpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.738' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.57933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒCû‚(:¸  ÉÈŠÓŽ»þ³ disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 262.92578125 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.sda.out‚ 262.92578125 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0# memory_percent‚88.1312370300293 disk.rbd1.out‚ 52.8154296875 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.dm-0.in‚ 83.52734375 disk.drbd1673.out‚0 network.sent‚ 1.003492e+07 disk.drbd1285.out‚0* proc.memory.percent‚11.829153696695965 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0( proc.memory.availableMB‚ 2916.8671875% disk./.usageGB‚1260.8515586853027 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0) disk./.usagePercent‚70.51341965847104 disk.drbd2111.in‚0 cpu‚25.324275241151632 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0" disk.dm-0.out‚31344.1005859375 network.recv‚599503 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0" proc.memory.rssMB‚ 2907.1328125% disk.nvme0n1.out‚31344.1005859375 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0rjžSžü‚(:“ ØÈŠÓ”µ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚266.114' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.70466666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}PWDý‚(:¹  ØÈŠÓóŸÑ³" disk.dm-0.out‚31344.2802734375 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.sda2.in‚0.03125 network.recv‚599569 disk.drbd1965.out‚0% disk.nvme0n1.out‚31344.2802734375 disk.drbd1673.in‚0 disk.dm-0.in‚ 83.52734375 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.sda2.out‚ 263.2109375 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 network.sent‚ 1.0034986e+07 disk.drbd1356.in‚0 disk.drbd2111.in‚0$ memory_percent‚88.13125292460123% disk./.usageGB‚1260.8515396118164 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2907.1328125) disk./.usagePercent‚70.51341859178183 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 52.8154296875 disk.drbd1285.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.829153696695965 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1817.out‚0 disk.sda1.in‚0 disk.sda.out‚ 263.2109375 disk.drbd1931.in‚0 disk.drbd1550.in‚0 cpu‚25.324217398169036 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0) proc.memory.availableMB‚ 2916.86328125•ùŸþ‚(:” çÈŠÓÿ÷®³ gpu.0.gpu‚69.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚301.807' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.60233333333335 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þY2kHÿ‚(:½  çÈŠÓ¼‰Ò³ disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚31344.90283203125 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 network.sent‚ 1.0035052e+07 network.recv‚599635 disk.sda.in‚0.03125 disk.sda.out‚ 263.71484375 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1550.out‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8517112731934 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2907.1328125) disk./.usagePercent‚70.51342819198467 disk.drbd2118.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 53.0654296875 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.829153696695965 disk.dm-0.in‚ 83.52734375 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 2917.11328125$ memory_percent‚88.13023567199707 proc.cpu.threads‚17# disk.dm-0.out‚31344.90283203125 disk.drbd1552.out‚0 disk.sda2.out‚ 263.71484375 disk.drbd1356.out‚0 cpu‚25.324133621102202 disk.sda2.in‚0.03125'$±ž€‚(:“ öÈŠÓõÙÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚283.151' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.38366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò-œF‚(:»  öÈŠÓ ƒð³ disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 264.00390625 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.829153696695965 disk.drbd1828.out‚0 proc.cpu.threads‚17 network.recv‚601587 disk.drbd2111.out‚0& disk.nvme0n1.out‚31345.06298828125 disk.drbd1584.in‚0 disk.drbd1289.in‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 53.0654296875 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 264.00390625 disk.drbd1938.out‚0$ memory_percent‚88.13126881917319' proc.memory.availableMB‚ 2916.859375 disk.drbd1510.out‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.51340771155195 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 83.52734375 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 cpu‚25.324024710489585 disk.dm-0.in‚ 83.52734375# disk.dm-0.out‚31345.06298828125 network.sent‚ 1.0044182e+07% disk./.usageGB‚1260.8513450622559 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0etÝ4‚/ _wandbruntime‚10300  _runtime‚10300"õêȃ & lr-AdamW/pg1‚7.817272670968434e-05 & lr-AdamW/pg2‚7.817272670968434e-05 trainer/global_step‚5649 % _timestamp‚1.7848495373842576e+09 _runtime‚10300.741519691 _step‚468ã;ÔÍ„Ç trainer/global_step‚5649 % _timestamp‚1.7848495373842576e+09 _runtime‚10300.741519691 _step‚468 & lr-AdamW/pg1‚7.817272670968434e-05 & lr-AdamW/pg2‚7.817272670968434e-05Ô_¤ö 4…/ _wandbruntime‚10300  _runtime‚10300Uów(†‚P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1627139083761664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1626496838007824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1627139109935104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5886359091648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101542097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52465429 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17210189 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445184153017344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5896232374272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452395780520448 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15058548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7211627503104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1633035316135936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445184521108480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17210536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5897577810432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452396199878656 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15058543 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1632383171671504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1627139109935104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7211678770176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49066604 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52475493 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5887703702934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1633035316135936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101532027 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49066598 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚347 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 trainer/global_step‚5649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ê„ZñÚ‡‡P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1633036687745536 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1626496812579856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5896232374272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1627139083761664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32268732 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52465429 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49066604 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1632384541710758 % _timestamp‚1.7848495373853657e+09 _runtime‚10300.742938574 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52475493 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5897577810432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452396199878656 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15058543 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1632383171671504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1627139109935104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7211678770176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49066604 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52475493 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5887703702934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1633035316135936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101532027 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49066598 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚347 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batcæÙëÜ3h_start/segment.large_pool.allocated‚1181 trainer/global_step‚5649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1627139083761664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1626496838007824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1627139109935104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5886359091648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101542097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52465429 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17210189 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445184153017344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5896232374272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452395780520448 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15058548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7211627503104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1633035316135936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445184521108480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17210536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101549476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101559560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1627417993052160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452476041665536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52474511 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5888724504686 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1627417993052160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5897254882304 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49074965 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1633316619551232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5898600325632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52484589 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1632664364286286 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1627418019225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1627418019225600 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32274658 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1633315247934464 _runtime‚10302.219394012 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15061188 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5898600325632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101559560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452475616023040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32274305 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655S{I.'Š‘N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1626775639781600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1626775614353632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52484589 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49074971 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1633315247934464 _runtime‚10302.219394012 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15061188 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5898600325632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101559560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452475616023040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32274305 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52474511 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17213470 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17213122 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚445263107474432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚445262739383296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49074971 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101549476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101559560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1627417993052160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452476041665536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52474511 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5888724504686 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1627417993052160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5897254882304 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49074965 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1633316619551232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5898600325632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52484589 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1632664364286286 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1627418019225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1627418019225600 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32274658 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1647031970216960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚456387549547008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7848496244033034e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10387.760888969 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49484948 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1641084548286464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5947421930496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49484948 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1640436926696624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 trainer/global_step‚5699 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7274131596288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1641084574459904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1647031970216960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚456387971002368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32547936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5948767366656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚449113839406080 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15190548 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5938807585742 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 _step‚473 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1647033341826560 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1641084574459904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5947421930496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5948767366656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17357393 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52931445 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52921381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102406329 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1647033341826560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7274078232064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5937462974456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52931445 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17357763 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1640436901268656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1646374364243112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528Ñý¶I (Ÿ‡P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17357763 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1640436901268656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1646374364243112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52921381 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1647031970216960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚456387549547008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7848496244033034e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10387.760888969 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49484948 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1641084548286464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5947421930496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49484948 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1640436926696624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 trainer/global_step‚5699 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7274131596288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1641084574459904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1647031970216960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚456387971002368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32547936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5948767366656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚449113839406080 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15190548 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5938807585742 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 _step‚473 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1647033341826560 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1641084574459904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5947421930496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5948767366656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17357393 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52931445 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52921381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102406329 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1647033341826560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7274078232064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5937462974456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52931445 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ù°WÄ4 / _wandbruntime‚10389  _runtime‚10389Yiö"à¡ŒN RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32553512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7275321020928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49493315 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5938483776096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102433862 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚363 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1646654186818528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1640715728470400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49493321 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1641363483750400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7275376475136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1641363457576960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5948444438528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102433862 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102423778 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5939828387494 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1640715703042432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52930463 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1647313273632256 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449192425772032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449192057680896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚5699 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5949789881856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17360692 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32553880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚456467378701824Z4HX'¢‘N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17360329 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1646655556857894 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚5699 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5949789881856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17360692 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32553880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚456467378701824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32553512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7275321020928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49493315 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5938483776096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102433862 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚363 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1646654186818528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1640715728470400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49493321 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1641363483750400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7275376475136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1641363457576960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5948444438528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102433862 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102423778 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5939828387494 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1640715703042432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52930463 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1647313273632256 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449192425772032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449192057680896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5999956922880 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7336719973888 _runtime‚10474.660639996 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚477 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1655030012811264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49903304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661028624297984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1655030038984704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5999956922880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15322543 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103280631 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5998611486720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17506065 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53387397 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661028624297984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1661029995907584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1655030012811264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49903298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5989911468550 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5988566857264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32828255 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53387397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17505712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7336668706816 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1660366926853974 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚5749 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1661029995907584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49903298 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚460379458319360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453043157703680 ûÕŸ (·‡P PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49903304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1654377015385424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49903304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661028624297984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1655030038984704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5999956922880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15322543 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103280631 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5998611486720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17506065 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53387397 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661028624297984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1661029995907584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1655030012811264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49903298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5989911468550 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5988566857264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32828255 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53387397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17505712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7336668706816 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1660366926853974 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚5749 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1661029995907584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49903298 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚460379458319360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453043157703680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15322548 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53377333 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53377333 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32828613 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5999956922880 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7336719973888 _runtime‚10474.660639996 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚477 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1655030012811264 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49911665 % _timestamp‚1.7848497127834795e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15325183 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚460459302905856 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103298080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1660646749429502 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1660645379390136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53386415 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32833804 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1655308922101760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103308164 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚453121375978496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _step‚478 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1655308948275200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6000979438080 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5749 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103308164 _runtime‚10476.141137193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7337973992960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0bi+ºº‘N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚357 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1661309927713280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚453121744069632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5999633994752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53386415 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15325188 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49911671 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1655308922101760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53396493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53396493 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5990932270302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49911665 % _timestamp‚1.7848497127834795e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15325183 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚460459302905856 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103298080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1660646749429502 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1660645379390136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53386415 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32833804 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1655308922101760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103308164 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitorðl]z.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚453121375978496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _step‚478 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1655308948275200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6000979438080 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5749 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103308164 _runtime‚10476.141137193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7337973992960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FžÄ‚(:“ æÊŠÓß©ß³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.11' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.03666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú•íFDÅ‚(:¹  æÊŠÓ ¹ƒ´( disk./.usagePercent‚70.5133818976732 disk.sda2.in‚0.03125 disk.drbd1289.out‚0* proc.memory.percent‚11.829153696695965 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 85.73828125 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 85.73828125 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 269.7265625 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 54.404296875 disk.drbd1552.out‚0 disk.drbd1673.out‚0# disk.dm-0.out‚31378.18701171875 disk.drbd2078.out‚0 cpu‚25.322839141279957 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0$ memory_percent‚92.08617210388184 network.recv‚613969 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8508834838867 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚31378.18701171875 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.sda2.out‚ 269.7265625 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd1510.out‚0 network.sent‚ 1.0194908e+07 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0) proc.memory.availableMB‚ 1944.90234375sF/ўƂ(:“ õÊŠÓµ‹×³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚256.237' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.41233333333335 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í¿pI3Ç‚(:¨  õÊŠÓÚíõ³ network.sent‚ 1.0256914e+07 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 1944.96484375 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.sda.out‚270.125 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 87.8984375 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1931.out‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚31384.4052734375 disk.drbd1226.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 54.4365234375 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚270.125 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0# memory_percent‚92.0859177907308* proc.memory.percent‚11.829153696695965% disk./.usageGB‚1260.8508834838867 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2907.1328125( disk./.usagePercent‚70.5133818976732 disk.drbd2111.in‚0 disk.drbd1289.out‚0" disk.dm-0.out‚31384.4052734375 disk.drbd1965.out‚0 disk.drbd1828.out‚0 network.recv‚616029 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.sda1.in‚0 disk.drbd1938.out‚0 cpu‚ 25.3227600091 Ý<žÈ‚(:“ „ËŠÓ´Ù½³ gpu.0.gpu‚55.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.769' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.25633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÔœCÉ‚(:¸  „ËŠÓÿœÛ³ disk.dm-0.in‚ 87.8984375 disk.drbd1356.in‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51337763091638" disk.dm-0.out‚31384.6240234375 disk.drbd1552.out‚0 proc.cpu.threads‚17 network.sent‚ 1.0265072e+07% disk./.usageGB‚1260.8508071899414 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 network.recv‚617157 cpu‚25.32265808044663 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 1944.90234375 disk.drbd1828.in‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 54.4365234375 disk.drbd1285.out‚0% disk.nvme0n1.out‚31384.6240234375 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.829153696695965$ memory_percent‚92.08617210388184 disk.sda.out‚ 270.40234375 disk.drbd1550.in‚0 disk.sda2.out‚ 270.40234375 disk.nvme0n1.in‚ 87.8984375" proc.memory.rssMB‚ 2907.1328125 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.drbd2118.out‚0 disk.sda.in‚0.03125’°~‡4Ê/ _wandbruntime‚10561  _runtime‚10561@aq©ÈË _step‚480 & lr-AdamW/pg1‚7.626612315683948e-05 & lr-AdamW/pg2‚7.626612315683948e-05 trainer/global_step‚5799 % _timestamp‚1.7848497983353162e+09 _runtime‚10561.692521092ú±ÿÍÌÇ _step‚480 & lr-AdamW/pg1‚7.626612315683948e-05 & lr-AdamW/pg2‚7.626612315683948e-05 trainer/global_step‚5799 % _timestamp‚1.7848497983353162e+09 _runtime‚10561.692521092à¨`Uœ4Í/ _wandbruntime‚10561  _runtime‚10561Ç1Áó(ÎP F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚5799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1675026649988608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1675025278379008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6041015351358 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1668975477336064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456972107910144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1674358119425582 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚481 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53833285 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1675025278379008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17653476 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1668975503509504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7399428780544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53843349 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104165003 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1668317104074224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15454548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1668317078646256 $ _timestamp‚1.784849798336294e+09 _runtime‚10561.696920751 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1668975477336064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1674356749386328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33108024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1668975503509504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50321648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6051146479104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15454543 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6039670740072 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50321654 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864‚“ÓWò φP ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1668975503509504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7399428780544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53843349 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104165003 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1668317104074224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15454548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1668317078646256 $ _timestamp‚1.784849798336294e+09 _runtime‚10561.696920751 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1668975477336064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1674356749386328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33108024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1`Z'Y668975503509504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50321648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6051146479104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15454543 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6039670740072 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50321654 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚5799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1675026649988608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1675025278379008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6041015351358 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1668975477336064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456972107910144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1674358119425582 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚481 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53833285 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1675025278379008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17653476 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653Ⴓ04Ð/ _wandbruntime‚10563  _runtime‚10563xa“ö'ÑŒN QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104172382 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50330021 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5799 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6040691541712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33113592 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1669254386626560 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104172382 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6050823550976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17656404 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1668595880420032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104182466 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104182466 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1675306581794304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚373 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848497998075979e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6052168994304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53842367 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6052168994304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1675306581794304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚10563.165170863 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚464451327713280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1668595905848000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15457183 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1669254386626560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33113219 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50330021 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53842367 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53852445 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6050823550976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50330015 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1675305210177536 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1674636571961744 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15457188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚457051062367232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457050694276096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 _step‚482 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1669254412800000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53852445 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7400633437184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6042036153110 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1669254412800000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50330015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17656036 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53842367 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6052168994304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1675306581794304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚10563.165170863 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚464451327713280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1668595905848000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15457183 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1669254386626560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33113219 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50330021 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53842367 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53852445 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6050823550976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50330015 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1675305210177536 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1674636571961744 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15457188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚457051062367232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457050694276096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 _step‚482 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1669254412800000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53852445 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7400633437184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6042036153110 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1669254412800000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50330015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17656036 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104172382 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50330021 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5799 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6040691541712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33113592 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1669254386626560 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104172382 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6050823550976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17656404 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1668595880420032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104182466 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104182466 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1675306581794304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚373 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398âv¡<µ4Ó/ _wandbruntime‚10563  _runtime‚10563l1 A Ô† /loss/FRt2/detection_loss‚1.5103241293512457 0loss/FBt2/detection_loss‚0.43386318419851716 trainer/global_step‚5799 _runtime‚10563.205194385 2loss/FBt2/assignment_loss‚0.010145024975348608 1loss/SRqqt2/detection_loss‚0.6811723548806075 epoch‚20 0loss/FRt2/assignment_loss‚0.6649256883679754 2loss/SRqqt1/assignment_loss‚0.5962747847415404 3loss/FBt1/assignment_loss‚0.0004170774809347473 _step‚483 1loss/SRqqt1/detection_loss‚0.9814900139847829 /loss/FBt1/detection_loss‚0.7605120992726263 $ _timestamp‚1.784849799848017e+09 %loss/total_loss‚8.407940519550824 0loss/FRt1/assignment_loss‚1.6896594687578537 2loss/SRqqt2/assignment_loss‚0.5158544321466816 /loss/FRt1/detection_loss‚0.5633022613927092}ɸÕ‹ 1loss/SRqqt1/detection_loss‚0.9814900139847829 /loss/FBt1/detection_loss‚0.7605120992726263 $ _timestamp‚1.784849799848017e+09 %loss/total_loss‚8.407940519550824 0loss/FRt1/assignment_loss‚1.6896594687578537 2loss/SRqqt2/assignment_loss‚0.5158544321466816 /loss/FRt1/detection_loss‚0.5633022613927092 /loss/FRt2/detection_loss‚1.5103241293512457 0loss/FBt2/detection_loss‚0.43386318419851716 trainer/global_step‚5799 _runtime‚10563.205194385 2loss/FBt2/assignment_loss‚0.010145024975348608 1loss/SRqqt2/detection_loss‚0.6811723548806075 epoch‚20 0loss/FRt2/assignment_loss‚0.6649256883679754 2loss/SRqqt1/assignment_loss‚0.5962747847415404 3loss/FBt1/assignment_loss‚0.0004170774809347473 _step‚483ã<Ü’¯žÖ‚(:“ “ËŠÓâγ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.732' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.91066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›suÞ@ׂ(:µ  “ËŠÓƒ¿í³( disk./.usagePercent‚70.5135975822303 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1828.in‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd1226.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.829153696695965 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.rbd1.out‚ 54.6005859375 disk.drbd1673.out‚0 disk.drbd2041.in‚0 cpu‚25.322578150707148 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1550.out‚0% disk./.usageGB‚1260.8547401428223 disk.drbd1550.in‚0 disk.sda.out‚ 270.828125 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1944.8515625 disk.nvme0n1.in‚ 87.8984375 disk.sda.in‚0.03125 disk.drbd2111.out‚0 network.recv‚617223 disk.sda2.out‚ 270.828125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚31388.95068359375 disk.drbd2078.in‚0 disk.drbd2078.out‚0 network.sent‚ 1.0265138e+07& disk.nvme0n1.out‚31388.95068359375 disk.drbd1965.out‚0$ memory_percent‚92.08637873331705 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 ,÷?Ø‚(:´  ¢ËŠÓðµ×³ disk.drbd1550.out‚0# disk.dm-0.out‚31389.07958984375 disk.drbd2078.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51359864891951 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0$ memory_percent‚92.08637873331705 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.sda.out‚ 271.21875 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8547592163086 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 cpu‚25.322523594769457 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1285.out‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.829153696695965 disk.drbd1817.in‚0 disk.sda2.out‚ 271.21875 disk.drbd1356.in‚0 network.recv‚617331 disk.drbd1285.in‚0& disk.nvme0n1.out‚31389.07958984375 disk.drbd2078.out‚0( proc.memory.availableMB‚ 1944.8515625 disk.rbd1.out‚ 54.6005859375 disk.nvme0n1.in‚ 87.8984375 disk.drbd1931.in‚0 network.sent‚ 1.0265246e+07 disk.drbd1938.in‚0 disk.drbd2041.in‚0®ˆeÎÙ‚(:” ¢ËŠÓÉ£µ³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.27®JïÑ9' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.75966666666667 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÙ¨5CÚ‚(:¸  ±ËŠÓ°üdz disk.drbd2041.out‚0& disk.nvme0n1.out‚31395.93310546875) disk./.usagePercent‚70.51360035562223 disk.dm-0.in‚ 87.8984375 disk.drbd1965.out‚0 disk.rbd1.out‚ 54.6328125 disk.drbd1584.out‚0$ memory_percent‚92.08842913309732" proc.memory.rssMB‚ 2907.1328125 cpu‚25.322398442992675 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8547897338867 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 network.recv‚620990 disk.drbd1673.in‚0# disk.dm-0.out‚31395.93310546875 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.829153696695965 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 271.56640625 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1944.34765625 network.sent‚ 1.0333463e+07 disk.drbd1289.out‚0 disk.sda.out‚ 271.56640625 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0™bð«žÛ‚(:“ ±ËŠÓ† ©³ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.069' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.35633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RàLBžÜ‚(:“ ÀËŠÓŽ÷²³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.585' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.86166666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›7û!AÝ‚(:¶  ÀËŠÓ­‚Ô³ disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 54.6328125 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 271.82421875 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚31396.42529296875" memory_percent‚92.086394627889 disk.drbd1673.in‚0 disk.sda2.out‚ 271.82421875 cpu‚25.322342971280268 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1944.84765625 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.dm-0.in‚ 87.8984375 disk.drbd2041.out‚0) disk./.usagePercent‚70.51356088812167# disk.dm-0.out‚31396.42529296875 disk.drbd2111.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0* proc.memory.percent‚11.829153696695965 network.sent‚ 1.0335739e+07 network.recv‚621122 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0" proc.memory.rssMB‚ 2907.1328125% disk./.usageGB‚1260.8540840148926kèt65Þ‚(:ª  ÏËŠÓÏÎ˳ disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.829153696695965 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 87.8984375 disk.dm-0.out‚ 31397.6328125 disk.drbd2111.out‚0 disk.rbd1.out‚ 54.796875 disk.drbd2078.out‚0 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 272.2265625 disk.drbd2078.in‚0 disk.drbd1356.in‚0 network.sent‚ 1.0335847e+07 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.51361699597382 disk.drbd1552.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1828.in‚0" disk.nvme0n1.out‚ 31397.6328125 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1944.84765625 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2907.1328125 cpu‚25.32231188682379 proc.cpu.threads‚17% disk./.usageGB‚1260.8550872802734 disk.sda2.in‚0.03125 disk.drbd1938.out‚0" memory_percent‚92.086394627889 network.recv‚621230 disk.drbd1510.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 272.2265625 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd2111.in‚0¦ §#žß‚(:“ ÏËŠÓꊯ³ gpu.0.gpu‚97.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.145' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.38166666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÏé¯4à/ _wandbruntime‚10648  _runtime‚106489BÈá & lr-AdamW/pg1‚7.562583669581282e-05 & lr-AdamW/pg2‚7.562583669581282e-05 trainer/global_step‚5849 % _timestamp‚1.7848498853830512e+09 _runtime‚10648.740282884 _step‚484Yz§{ÍâÇ & lr-AdamW/pg1‚7.562583669581282e-05 & lr-AdamW/pg2‚7.562583669581282e-05 trainer/global_step‚5849 % _timestamp‚1.7848498853830512e+09 _runtime‚10648.740282884 _step‚484äÝÛ*4ã/ _wandbruntime‚10648  _runtime‚10648}üR=(äP OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6102336035328 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1682920941860864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1682257167335056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54289237 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54289237 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7462042140672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1682920968034304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15586548 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54299301 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6100990599168 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1689023304069632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6102336035328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6100990599168 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1688349311997190 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33387538 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105029235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50739998 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50740004 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 $ _timestamp‚1.784849885384911e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50740004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6090774622880 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50739998 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1689021932460032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54299301 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1689021932460032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105039305 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚10648.742459147 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚350 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460901426207744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7462095504896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1682257192763024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468363468348416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460901794298880 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1689021932460032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54299301 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1689021932460032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105039305 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚10648.742459147 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚350 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460901426207744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7462095504896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1682257192763024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468363468348416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460901794298880 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105046684 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460980012573696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15589183 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.cur^z¹prent‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6103358550528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7463344993792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6091795424520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1689301864258560 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50748365 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105056768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1688629134572718 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105046684 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54298319 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33393100 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17803917 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50748371 _runtime‚10650.221458179 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1689303235875328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6102013107200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17804270 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6093140035918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105056768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1688629134572718 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105046684 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54298319 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33393100 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17803917 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50748371 _runtime‚10650.221458179 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1689303235875328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6102013107200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17804270 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6093140035918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105046684 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460980012573696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15589183 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6103358550528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7463344993792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6091795424520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1689301864258560 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0抃è4ë/ _wandbruntime‚10650  _runtime‚10650…¶(@ ì„ % _timestamp‚1.7848498868980196e+09 _runtime‚10650.255222419 2loss/SRqqt2/assignment_loss‚0.5095949131368929 2loss/FBt2/assignment_loss‚0.061176196444766476 .loss/FRt2/detection_loss‚1.461966908996075 /loss/FBt1/detection_loss‚0.9049728912456582 %loss/total_loss‚8.841743987956024 epoch‚20 _step‚487 0loss/FRt2/assignment_loss‚0.7325176652950076 2loss/SRqqt1/assignment_loss‚0.6293121951895765 2loss/FBt1/assignment_loss‚0.002633438977546495 trainer/global_step‚5849 0loss/FRt1/assignment_loss‚1.7338763190095559 /loss/FRt1/detection_loss‚0.5556642961831529 1loss/SRqqt1/detection_loss‚0.9769376085240589 1loss/SRqqt2/detection_loss‚0.6167979012990807 /loss/FBt2/detection_loss‚0.6562936536546526·à9"í‰ /loss/FRt1/detection_loss‚0.5556642961831529 1loss/SRqqt1/detection_loss‚0.9769376085240589 1loss/SRqqt2/detection_loss‚0.6167979012990807 /loss/FBt2/detection_loss‚0.6562936536546526 % _timestamp‚1.7848498868980196e+09 _runtime‚10650.255222419 2loss/SRqqt2/assignment_loss‚0.5095949131368929 2loss/FBt2/assignment_loss‚0.061176196444766476 .loss/FRt2/detection_loss‚1.461966908996075 /loss/FBt1/detection_loss‚0.9049728912456582 %loss/total_loss‚8.841743987956024 epoch‚20 _step‚487 0loss/FRt2/assignment_loss‚0.7325176652950076 2loss/SRqqt1/assignment_loss‚0.6293121951895765 2loss/FBt1/assignment_loss‚0.002633438977546495 trainer/global_step‚5849 0loss/FRt1/assignment_loss‚1.7338763190095559çwÅ )î‚(:’ íËŠÓÁÝ׳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚154.97' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚51.65666666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú!˹Dï‚(:¹  íËŠÓœ¼ö³$ disk./.usageGB‚1260.855525970459 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51364152982552 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.sda.out‚ 272.86328125 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 272.86328125 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0) proc.memory.availableMB‚ 1944.56640625 network.recv‚624505 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.rbd1.out‚ 54.8291015625 cpu‚25.322130824935925 network.sent‚ 1.0378479e+07 disk.nvme0n1.in‚ 87.8984375# memory_percent‚92.0875390370687 disk.dm-0.in‚ 87.8984375# disk.dm-0.out‚31398.11865234375 disk.drbd1356.out‚0& disk.nvme0n1.out‚31398.11865234375 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.829153696695965 disk.drbd1931.out‚0 disk.drbd1965.out‚0büržð‚(:“ üˊӭ׳ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚291.134' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.04466666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06?™Bñ‚(:·  üËŠÓ·˜ô³ network.sent‚ 1.0378587e+07* proc.memory.percent‚11.829153696695965 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0# memory_percent‚92.0875390370687 disk.rbd1.out‚ 54.8291015625 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 cpu‚25.322076595639327 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 273.2734375 disk.dm-0.in‚ 87.8984375 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1944.56640625& disk.nvme0n1.out‚31398.29052734375 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 network.recv‚624613# disk.dm-0.out‚31398.29052734375 disk.nvme0n1.in‚ 87.8984375 disk.drbd1552.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51364152982552 disk.sda2.out‚ 273.2734375" proc.memory.rssMB‚ 2907.1328125$ disk./.usageGB‚1260.855525970459 disk.drbd1965.out‚0 disk.drbd2041.in‚0‡ÝC7Cò‚(:¸  ‹ÌŠÓÕ½Ö³ disk.sda.in‚0.03125 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1944.56640625* proc.memory.percent‚11.829153696695965 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 273.6328125# disk.dm-0.out‚31398.74658203125 disk.drbd1285.out‚0 disk.drbd1550.in‚0 cpu‚25.321976978715444 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 network.sent‚ 1.0379064e+07 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 54.9931640625 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1510.out‚0# memory_percent‚92.0875390370687 disk.drbd1584.in‚0 disk.sda2.out‚ 273.6328125) disk./.usagePercent‚70.51365582346085 disk.drbd1965.out‚0& disk.nvme0n1.out‚31398.74658203125 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1226.out‚0 network.recv‚625509% disk./.usageGB‚1260.8557815551758 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1356.out‚0 disk.drbd1828.in‚0ìª-©“ó‚(:ˆ ‹ÌŠÓïê·³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.665' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿û–¸Ÿô‚(:” šÌŠÓ¨Ãß³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.016' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.00533333333335 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰¥VÊDõ‚(:¹  šÌŠÓÖë„´ disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.829153696695965 proc.cpu.threads‚17% disk./.usageGB‚1260.8563346862793 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 55.2587890625 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd2111.out‚0# memory_percent‚92.0875390370687 disk.drbd1673.out‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚31398.98095703125 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 1944.56640625 disk.drbd1817.out‚0& disk.nvme0n1.out‚31398.98095703125 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.out‚ 274.05859375 disk.sda2.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 274.05859375 disk.drbd1550.out‚0 cpu‚25.323503569076248 network.recv‚625575 disk.dm-0.in‚ 87.8984375 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 network.sent‚ 1.037913e+07) disk./.usagePercent‚70.51368675744779 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0¢æ4ö/ _wandbruntime‚10721  _runtime‚10721û7ù÷Ëu ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8361299633979797 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927783012390137 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7273834347724915 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7308064103126526 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5965010523796082 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9752354025840759 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 'particle/f_score‚0.6476436257362366 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6405724883079529 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.828509509563446 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6339529156684875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9236628413200378 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6440556645393372 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 _step‚488 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4322698414325714 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7359839081764221 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22755296528339386 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870508909225464 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.804172933101654 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29625192284584045 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958144426345825 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7149476408958435 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03448650613427162 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021536018699407578 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6872414350509644 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17815928161144257 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11456543952226639 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027149749919772148 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739557862281799 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6321018934249878 particle/accuracy_1_of_2‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5611768960952759 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21491922438144684 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23066210746765137 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8061147332191467 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26007840037345886 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.253114400228085 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7081446647644043 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05424290895462036 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2698543667793274 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20219136774539948 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5538461208343506 trainer/global_step‚5879 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3773333728313446 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29536035656929016 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3754379153251648 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 *jet/accuracy_2_of_3‚0.5618047118186951 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.11347367614507675 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9970135688781738 7 Purity/*FRt0SRqqt1FBt/FBt_puri¹`ô‹P"ty‚0.9991288781166077 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3045787513256073 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10791884362697601 7validation_average_jet_accuracy‚0.38060829043388367 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27386629581451416 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6746771931648254 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41096144914627075 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29482999444007874 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2824692726135254 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8631619215011597 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22551141679286957 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2256648987531662 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6643515229225159 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6608428955078125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6348901391029358 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2824692726135254 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34493082761764526 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8273038268089294 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6011000871658325 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7692307829856873 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30990543961524963 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7556087374687195 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12900207936763763 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028790561482310295 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04640212655067444 particle/accuracy_3_of_4‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9332876205444336 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4565306603908539 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7372717261314392 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07447508722543716 +particle/specificity‚0.9063298106193542 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3027271628379822 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.243193119764328 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28174683451652527 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5176210999488831 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8688774108886719 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032162945717573166 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021536018699407578 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6031530499458313 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.418982595205307 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8262626528739929 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7642397284507751 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +jet/accuracy_3_of_3‚0.07373758405447006 _runtime‚10721.947217661 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.672826886177063 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8033841848373413 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03705483675003052 $particle/accuracy‚0.849853515625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24128593504428864 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7845599055290222 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7692015767097473 % _timestamp‚1.7848499585890923e+09 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 epoch‚20 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6898336410522461 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739557862281799 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038109611719846725 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7247439026832581 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6138157248497009 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6352834105491638 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 +jet/accuracy_2_of_2‚0.15460248291492462 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.629072368144989 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11390642821788788 *jet/accuracy_1_of_2‚0.7712708115577698 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7302011251449585 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.727998673915863 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7257570624351501 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19782818853855133 *particle/sensitivity‚0.643102765083313 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6056694388389587 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4516237676143646 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7634691596031189 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33835965394973755 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18098972737789154 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8268939256668091 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09985842555761337 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6449800729751587 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905175566673279 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 /particle/accuracy_4_of_4‚0.9560583829879761 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6844829320907593 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893012046813965 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21334992349147797 /particle/accuracy_3_of_3‚0.9993678331375122 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7275780439376831 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 )jet/accuracy_1_of_1‚0.343858003616333 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9132272601127625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6372728943824768 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6352834105491638 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125"„ò Ö:øÐu 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28174683451652527 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5176210999488831 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8688774108886719 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032162945717573166 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021536018699407578 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6031530499458313 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.418982595205307 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8262626528739929 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7642397284507751 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +jet/accuracy_3_of_3‚0.07373758405447006 _runtime‚10721.947217661 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.672826886177063 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8033841848373413 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03705483675003052 $particle/accuracy‚0.849853515625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24128593504428864 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7845599055290222 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7692015767097473 % _timestamp‚1.7848499585890923e+09 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 epoch‚20 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6898336410522461 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739557862281799 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038109611719846725 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7247439026832581 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6138157248497009 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6352834105491638 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 +jet/accuracy_2_of_2‚0.15460248291492462 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.629072368144989 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11390642821788788 *jet/accuracy_1_of_2‚0.7712708115577698 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7302011251449585 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.727998673915863 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7257570624351501 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19782818853855133 *particle/sensitivity‚0.643102765083313 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6056694388389587 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4516237676143646 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7634691596031189 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33835965394973755 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18098972737789154 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8268939256668091 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09985842555761337 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6449800729751587 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905175566673279 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 /particle/accuracy_4_of_4‚0.9560583829879761 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6844829320907593 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893012046813965 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21334992349147797 /particle/accuracy_3_of_3‚0.9993678331375122 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7275780439376831 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 )jet/accuracy_1_of_1‚0.343858003616333 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9132272601127625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6372728943824768 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6352834105491638 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8361299633979797 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927783012390137 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7273834347724915 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7308064103126526 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5965010523796082 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9752354025840759 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 'particle/f_score‚0.6476436257362366 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6405724883079529 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.828509509563446 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6339529156684875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9236628413200378 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6440556645393372 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 _step‚488 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4322698414325714 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7359839081764221 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22755296528339386 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870508909225464 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.804172933101654 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29625192284584045 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958144426345825 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7149476408958435 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03448650613427162 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021536018699407578 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6872414350509644 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17815928161144257 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11456543952226639 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027149749919772148 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739557862281799 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6321018934249878 particle/accuracy_1_of_2‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5611768960952759 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21491922438144684 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23066210746765137 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8061147332191467 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26007840037345886 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.253114400228085 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7081446647644043 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05424290895462036 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2698543667793274 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20219136774539948 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5538461208343506 trainer/global_step‚5879 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3773333728313446 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29536035656929016 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3754379153251648 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 *jet/accuracy_2_of_3‚0.5618047118186951 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.11347367614507675 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9970135688781738 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3045787513256073 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10791884362697601 7validation_average_jet_accuracy‚0.38060829043388367 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27386629581451416 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6746771931648254 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41096144914627075 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29482999444007874 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2824692726135254 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8631619215011597 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22551141679286957 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2256648987531662 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6643515229225159 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6608428955078125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6348901391029358 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2824692726135254 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34493082761764526 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8273038268089294 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6011000871658325 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7692307829856873 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30990543961524963 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7556087374687195 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12900207936763763 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028790561482310295 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04640212655067444 particle/accuracy_3_of_4‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9332876205444336 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4565306603908539 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7372717261314392 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07447508722543716 +particle/specificity‚0.9063298106193542 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3027271628379822 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.243193119764328 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1èK‹Ècù‚(:’ ©ÌŠÓ©û·³ gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚58.0 gpu.0.powerWatts‚105.929' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.309666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£µˆAú‚(:¶  ©ÌŠÓ¸Çܳ disk.drbd1289.out‚0# proc.memory.rssMB‚ 3212.70703125 network.sent‚ 1.0444271e+07 network.recv‚627290 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.sda.out‚ 274.53125 disk.drbd1965.in‚0) proc.memory.availableMB‚ 2611.11328125) disk./.usagePercent‚70.51389604186966 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 55.3212890625 cpu‚25.326191094019876 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0$ memory_percent‚89.37535285949707 disk.drbd1226.out‚0# disk.dm-0.out‚31399.32861328125 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 274.53125% disk./.usageGB‚1260.8600769042969 disk.drbd1552.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚13.072538375854492& disk.nvme0n1.out‚31399.32861328125 disk.drbd1584.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0ò©ÃS”û‚(:‰ ¸ÌŠÓԾѳ gpu.0.gpu‚85.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚305.013' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.671 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ë!Bü‚(:·  ¸ÌŠÓþÔï³ proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd2118.in‚0( proc.memory.availableMB‚ 2917.0703125" disk.dm-0.out‚32369.4208984375 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 cpu‚25.325063261555364 disk.rbd1.out‚ 55.3212890625$ disk./.usageGB‚1260.859962463379 disk.drbd1938.in‚0$ memory_percent‚88.13041051228842 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 274.96484375* proc.memory.percent‚11.829153696695965 network.recv‚628897 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 274.96484375 network.sent‚ 1.0500583e+07) disk./.usagePercent‚70.51388964173442 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 87.8984375% disk.nvme0n1.out‚32369.4208984375 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0" proc.memory.rssMB‚ 2907.1328125ÂçVñ“ý‚(:ˆ ÇÌŠÓÅñ´³ gpu.0.gpu‚68.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚107.151' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.717 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@×ìAþ‚(:¶  ÇÌŠÓ¢ŠØ³ disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51388964173442 disk.drbd1356.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.829153696695965 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 network.sent‚ 1.0500691e+07 disk.nvme0n1.in‚ 87.8984375 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 55.4853515625$ memory_percent‚88.13041051228842" proc.memory.rssMB‚ 2907.1328125 disk.drbd1817.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 275.1796875 disk.drbd1289.in‚0# disk.dm-0.out‚32370.57861328125$ disk./.usageGB‚1260.859962463379 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 cpu‚25.32495980494159 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0 disk.sda.out‚ 275.1796875 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd2041.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚32370.57861328125 network.recv‚629005 disk.drbd1285.out‚0( proc.memory.availableMB‚ 2917.0703125 disk.drbd1552.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17M7Šë4ÿ/ _wandbruntime‚10760  _runtime‚10760Ôìñ1Ȁ % _timestamp‚1.7848499969447074e+09 _runtime‚10760.302893315 _step‚489 & lr-AdamW/pg1‚7.498331388594989e-05 & lr-AdamW/pg2‚7.498331388594989e-05 trainer/global_step‚5899ÆIzûÍÇ trainer/global_step‚5899 % _timestamp‚1.7848499969447074e+09 _runtime‚10760.302893315 _step‚489 & lr-AdamW/pg1‚7.498331388594989e-05 & lr-AdamW/pg2‚7.498331388594989e-05éÝ™µ4‚/ _wandbruntime‚10760  _runtime‚10760ê)ǶúƒP MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51247786 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1705584881704960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚10760.302974088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33701517 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51247780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 $ _timestamp‚1.784849996945454e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54802565 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17975778 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472411702038016 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1704903744520026 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1705586253314560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54802565 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7531958019584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1698756257175024 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1699427109283840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6159144030720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33701213 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚5899 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51247780 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1705586253314560 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106050351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17976077 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚490 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 408872448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6159144030720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1705586253314560 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106050351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17976077 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚490 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 408872448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6159144030720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51247786 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1705584881704960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚10760.302974088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33701517 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51247780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 $ _timestamp‚1.784849996945454e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54802565 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17975778 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472411702038016 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1704903744520026 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1705586253314560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54802565 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7531958019584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1698756257175024êïúäF4…/ _wandbruntime‚10761  _runtime‚10761J­G'†ŒN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51256153 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1705184937134808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54811661 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1699706018574336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6149852758040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6158821102592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚472491983666176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464958330384384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚491 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17978716 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106067814 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7533285190656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 trainer/global_step‚5899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106057730 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848499984218783e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1699706018574336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33706791 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6148508146642 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6160166545920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17979015 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54801583 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106057730 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1705864813503488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51256153 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51256147 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 408865280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54801583 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7533244416512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1705866185120256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1705866185120256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51256147 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1699035084376768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1699705992400896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6160166545920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1699705992400896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15728075 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51256147 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 408865280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54801583 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7533244416512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1705866185120256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1705866185120256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51256147 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1699035084376768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1699705992400896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6160166545920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1699705992400896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15728075 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsM½ŽÆl™onitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1705184937134808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54811661 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1699706018574336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6149852758040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6158821102592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚472491983666176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464958330384384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚491 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17978716 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106067814 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7533285190656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 trainer/global_step‚5899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106057730 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848499984218783e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1699706018574336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33706791 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6148508146642 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6160166545920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17979015 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54801583 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106057730 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1705864813503488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51256153 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 Q’‚(:³  ƒÍŠÓŒˆè³) proc.memory.availableMB‚ 2916.47265625 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0* proc.memory.percent‚11.829153696695965 network.sent‚ 1.0574671e+07 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd1938.out‚0$ memory_percent‚88.13284238179526 disk.drbd2111.out‚0 disk.rbd1.out‚ 55.7978515625 disk.drbd1285.out‚0 disk.drbd1673.in‚0 network.recv‚633020 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0" disk.dm-0.out‚33342.2587890625% disk.nvme0n1.out‚33342.2587890625 disk.drbd1584.in‚0$ disk./.usageGB‚1260.856185913086 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 277.234375 disk.drbd1931.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0" proc.memory.rssMB‚ 2907.1328125 disk.dm-0.in‚ 87.8984375 disk.drbd1828.out‚0 disk.sda2.out‚ 277.234375 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0) disk./.usagePercent‚70.51367843727199 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 87.8984375 cpu‚25.32463679461662 disk.sda1.out‚0 disk.drbd1931.in‚0qhûŸ“‚(:” ’ÍŠÓò³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚328.031' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚109.34366666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠšC”‚(:¸  ’ÍŠÓóõÖ³) disk./.usagePercent‚70.51367843727199 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 network.sent‚ 1.0574779e+07 disk.drbd1289.out‚0 disk.drbd1289.in‚0" disk.dm-0.out‚33342.3955078125 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 55.9775390625 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.sda2.out‚ 277.73828125% disk.nvme0n1.out‚33342.3955078125 disk.drbd1226.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd2078.in‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd2041.out‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 2916.47265625 network.recv‚633128 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1510.in‚0$ memory_percent‚88.13284238179526 disk.sda.out‚ 277.73828125 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0$ disk./.usageGB‚1260.856185913086 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.829153696695965 cpu‚25.324580418165183 disk.drbd1552.in‚0 disk.drbd1931.in‚0ž1”¤ž•‚(:“ ¡ÍŠÓäÞÒ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚265.535' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.51166666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼eO¿:–‚(:¯  ¡ÍŠÓæÍõ³ disk.drbd1285.out‚0 cpu‚25.324476425288687 network.recv‚634009" proc.memory.rssMB‚ 2907.1328125 disk.sda.out‚ 278.0546875 disk.sda2.in‚0.03125 disk.drbd1552.in‚0) proc.memory.availableMB‚ 2916.72265625$ disk./.usageGB‚1260.856185913086 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0$ memory_percent‚88.13182512919107 disk.drbd1965.out‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51367843727199 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 network.sent‚ 1.057576e+07* proc.memory.percent‚11.829153696695965 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.dm-0.out‚ 33342.6796875 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd2041.in‚0" disk.nvme0n1.out‚ 33342.6796875 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.drbd1817.out‚0 disk.sda2.out‚ 278.0546875 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 56.0400390625 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚07xb4—/ _wandbruntime‚10847  _runtime‚10847²ÙÍ3Ș & lr-AdamW/pg1‚7.433863894727014e-05 & lr-AdamW/pg2‚7.433863894727014e-05 trainer/global_step‚5949 % _timestamp‚1.7848500840924227e+09 _runtime‚10847.449657905 _step‚493,V2ꪂ & lr-AdamW/pg2‚7.433863894727014e-05 trainer/global_step‚5949 % _timestamp‚1.7848500840924227e+09 _runtime‚10847.449657905 _step‚493 & lr-AdamW/pg1‚7.433863894727014e-05í·?Ñ4š/ _wandbruntime‚10847  _runtime‚10847©}Ó(›‚P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51666136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55248453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _runtime‚10847.454973014 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1719582907395584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6198591227810 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33982378 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55258517 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55248453 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1718896307130888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848500840935087e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6199935839096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51666130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476404048827904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7595039876096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15857435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476404470283264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚494 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51666130 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468809430407168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1712696371291792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6210333586944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51666136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15857440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1719581535785984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6210333586944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6208988150784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1718894937091634 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106924653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18124596 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7594986511872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1712696345863824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1719582907395584 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1713372573808640û¢„F œ‡P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15857440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1719581535785984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6210333586944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6208988150784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1718894937091634 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106924653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18124596 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7594986511872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1712696345863824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1719582907395584 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1713372573808640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak¶HJÇ‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51666136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55248453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _runtime‚10847.454973014 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1719582907395584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6198591227810 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33982378 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55258517 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55248453 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1718896307130888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848500840935087e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6199935839096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51666130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476404048827904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7595039876096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15857435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476404470283264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚494 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51666130 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468809430407168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1712696371291792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6210333586944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51666136î2}œ4/ _wandbruntime‚10848  _runtime‚10848µˆØÊ'žŒN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476484303155200 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1713651456925696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51674503 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6210010658816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33987941 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1719861467584512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1719176129706416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1719862839201280 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚5949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55267613 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7596235122176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1719174759667050 _step‚495 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51674503 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18127516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1712975173065568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1719862839201280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33987591 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51674497 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15860080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15860075 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106932032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55267613 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1712975147637600 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6200956640848 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106932032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106942116 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚5949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55267613 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7596235122176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1719174759667050 _step‚495 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51674503 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18127516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1712975173065568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1719862839201280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33987591 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51674497 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15860080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15860075 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106932032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55267613 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1712975147637600 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6200956640848 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106932032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106942116 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1713651456925696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51674503 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6210010658816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33987941 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1719861467584512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1719176129706416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1719862839201280 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904ï¿Íå›4 / _wandbruntime‚10848  _runtime‚10848ÙÒ»¡ /loss/FRt1/detection_loss‚0.5201130743420934 %loss/total_loss‚7.954249001004495 trainer/global_step‚5949 /loss/FRt1/assignment_loss‚1.668947825333436 3loss/FBt1/assignment_loss‚0.0017068628236531578 1loss/FBt2/assignment_loss‚0.03452526451586866 /loss/FBt1/detection_loss‚0.6470171973907521 /loss/FBt2/detection_loss‚0.2505219229847856 epoch‚21 _step‚496 0loss/FRt2/assignment_loss‚0.7427507230138357 1loss/SRqqt2/detection_loss‚0.6183883636715287 _runtime‚10848.963922123 /loss/SRqqt1/assignment_loss‚0.5652114781207 2loss/SRqqt2/assignment_loss‚0.5025473604269224 /loss/FRt2/detection_loss‚1.4470586520506366 1loss/SRqqt1/detection_loss‚0.9554602763302825 % _timestamp‚1.7848500856067102e+09üä7 ¢† /loss/FRt1/detection_loss‚0.5201130743420934 %loss/total_loss‚7.954249001004495 trainer/global_step‚5949 /loss/FRt1/assignment_loss‚1.668947825333436 3loss/FBt1/assignment_loss‚0.0017068628236531578 1loss/FBt2/assignment_loss‚0.03452526451586866 /loss/FBt1/detection_loss‚0.6470171973907521 /loss/FBt2/detection_loss‚0.2505219229847856 epoch‚21 _step‚496 0loss/FRt2/assignment_loss‚0.7427507230138357 1loss/SRqqt2/detection_loss‚0.6183883636715287 _runtime‚10848.963922123 /loss/SRqqt1/assignment_loss‚0.5652114781207 2loss/SRqqt2/assignment_loss‚0.5025473604269224 /loss/FRt2/detection_loss‚1.4470586520506366 1loss/SRqqt1/detection_loss‚0.9554602763302825 % _timestamp‚1.7848500856067102e+09ð’HCÞž£‚(:“ °ÍŠÓΡ½³ gpu.0.gpu‚78.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚299.428' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.80933333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜™p­3¤‚(:¨  °ÍŠÓÌÌݳ) proc.memory.availableMB‚ 2916.23046875 network.recv‚634075 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.829153696695965 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 56.0400390625 disk.nvme0n1.in‚ 87.8984375 disk.drbd1584.in‚0 disk.drbd1289.out‚0$ memory_percent‚88.13382784525552 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1938.out‚0# disk.nvme0n1.out‚33342.89453125 disk.drbd1550.out‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51367737058278 disk.dm-0.out‚33342.89453125% disk./.usageGB‚1260.8561668395996 disk.sda2.out‚278.25 disk.dm-0.in‚ 87.8984375 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda.out‚278.25 disk.drbd1510.in‚0 cpu‚25.32446657967642 network.sent‚ 1.0575826e+07 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0" proc.memory.rssMB‚ 2907.1328125 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0Sc@Ÿ¥‚(:” ¿ÍŠÓíþ³³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚301.897' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.63233333333332 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬iàŸØ¦‚(:¸  ¿ÍŠÓËöÓ³% disk./.usageGB‚1260.8525428771973 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 cpu‚25.32436455197903 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.829153696695965% disk.nvme0n1.out‚33343.2802734375) proc.memory.availableMB‚ 2916.43359375 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 network.recv‚634183 disk.drbd1550.out‚0 disk.drbd1817.in‚0" disk.dm-0.out‚33343.2802734375 disk.drbd1965.in‚0" proc.memory.rssMB‚ 2907.1328125 disk.sda1.in‚0 disk.sda2.out‚ 278.55859375 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 56.2275390625 disk.nvme0n1.in‚ 87.8984375 disk.drbd1550.inw"DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1733578189867008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7657652073472 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1732887499702496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52084486 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1733579561476608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107788885 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107798955 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1727318012160000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55714469 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6261523143168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18272611 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1733579561476608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1727318038333440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6261523143168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1726636459980592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1733578189867008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55704405 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6249695110618 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚357 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52084486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚10934.232443621 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107788885 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107798955 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1727318012160000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55714469 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6261523143168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18272611 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1733579561476608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1727318038333440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6261523143168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1726636459980592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1733578189867008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55704405 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6249695110618 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚357 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52084486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚10934.232443621 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1733578189867008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7657652073472 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1732887499702496ò‹G@³‚(:µ  ûÍŠÓªÌã³ disk.nvme0n1.in‚ 87.8984375 disk.drbd2118.out‚0 disk.dm-0.in‚ 87.8984375% disk./.usageGB‚1260.8564376831055 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 56.423828125 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 network.sent‚ 1.065934e+07 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0% proc.memory.availableMB‚ 1950.3125 disk.drbd1285.out‚0) disk./.usagePercent‚70.51369251756948& disk.nvme0n1.out‚33344.55615234375 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1289.out‚0 network.recv‚649434 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1510.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.06415812174478 disk.drbd1226.in‚0# disk.dm-0.out‚33344.55615234375 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.835050582885742 disk.drbd1510.in‚0 disk.sda.out‚ 280.08984375 disk.sda2.in‚0.03125 disk.sda2.out‚ 280.08984375 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 cpu‚25.32409392252626 disk.drbd2041.in‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2908.58203125 disk.drbd1550.out‚0vt…ž´‚(:“ ûÍŠÓ¹Þ³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚258.152' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.05066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“'¼4µ/ _wandbruntime‚10935  _runtime‚10935[<=ö† ¶‹N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6252060523656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52092847 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1727596947623936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1726915236326400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7658856658432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1733165952238658 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15992080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6261200215040 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1727596947623936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52092853 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6262545658368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 trainer/global_step‚5999 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55723565 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784850172349817e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1727596921450496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34267219 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480475823638016 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107816418 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107816418 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52092847 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107806334 _step‚499 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1733859493282304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107806334…—'·N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6262545658368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 trainer/global_step‚5999 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55723565 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784850172349817e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1727596921450496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34267219 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480475823638016 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107816418 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107816418 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52092847 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107806334 _step‚499 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1733859493282304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107806334 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6252060523656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52092847 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1727596947623936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1726915236326400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7658856658432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1733165952238658 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15992080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6261200215040 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1727596947623936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52092853ó•4¸/ _wandbruntime‚10935  _runtime‚10935Ì9+¢¹‚ .loss/FRt2/detection_loss‚1.474705105199857 /loss/FBt2/detection_loss‚0.4504177678600447 epoch‚21 trainer/global_step‚5999 % _timestamp‚1.7848501723864453e+09 0loss/FRt1/assignment_loss‚1.7893869127529194 0loss/FRt2/assignment_loss‚0.7294166967204717 2loss/SRqqt2/assignment_loss‚0.5735761063688187 1loss/SRqqt2/detection_loss‚0.6628378475886513 %loss/total_loss‚9.248889275399979 _runtime‚10935.743690499 2loss/SRqqt1/assignment_loss‚0.6445036434767999 3loss/FBt1/assignment_loss‚0.0033802177165461623 /loss/FBt2/assignment_loss‚0.041480570072962 /loss/FBt1/detection_loss‚1.2991507863280949 /loss/FRt1/detection_loss‚0.5914164190540763 1loss/SRqqt1/detection_loss‚0.9886172022607367 _step‚500óä& º‡ 0loss/FRt2/assignment_loss‚0.7294166967204717 2loss/SRqqt2/assignment_loss‚0.5735761063688187 1loss/SRqqt2/detection_loss‚0.6628378475886513 %loss/total_loss‚9.248889275399979 _runtime‚10935.743690499 2loss/SRqqt1/assignment_loss‚0.6445036434767999 3loss/FBt1/assignment_loss‚0.0033802177165461623 /loss/FBt2/assignment_loss‚0.041480570072962 /loss/FBt1/detection_loss‚1.2991507863280949 /loss/FRt1/detection_loss‚0.5914164190540763 1loss/SRqqt1/detection_loss‚0.9886172022607367 _step‚500 .loss/FRt2/detection_loss‚1.474705105199857 /loss/FBt2/detection_loss‚0.4504177678600447 epoch‚21 trainer/global_step‚5999 % _timestamp‚1.7848501723864453e+09 0loss/FRt1/assignment_loss‚1.7893869127529194ôÿ*AXŸ»‚(:” ŠÎŠÓزȳ gpu.0.gpu‚46.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚310.232' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.41066666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Y3BB¼‚(:·  ŠÎŠÓ¥ùæ³ network.recv‚651150% disk./.usageGB‚1260.8555717468262 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚33344.76318359375# proc.memory.rssMB‚ 2908.58203125 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.51364408987962 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1356.out‚0 network.sent‚ 1.070793e+07 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0# memory_percent‚92.0633316040039 disk.drbd1550.out‚0 disk.dm-0.in‚ 87.8984375 disk.rbd1.out‚ 56.486328125 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0# disk.dm-0.out‚33344.76318359375 disk.drbd1828.in‚0' proc.memory.availableMB‚ 1950.515625 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 280.45703125 disk.sda2.out‚ 280.45703125 disk.drbd1965.in‚0 cpu‚25.324038125452518 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.835050582885742öZ8½‚(:­  ™ÎŠÓüÞý³ disk.dm-0.out‚ 33345.2890625 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0* proc.memory.percent‚11.835050582885742 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0" disk.nvme0n1.out‚ 33345.2890625 disk.drbd1550.out‚0 proc.cpu.threads‚17 network.recv‚655110 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 cpu‚25.32391356156652 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51375289217845 disk.sda.out‚ 280.8046875 disk.sda2.out‚ 280.8046875 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2908.58203125 network.sent‚ 1.0758363e+07% disk./.usageGB‚1260.8575172424316 disk.drbd1828.in‚0' proc.memory.availableMB‚ 1950.515625 disk.drbd2078.in‚0 disk.rbd1.out‚ 56.486328125 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0# memory_percent‚92.0633316040039 disk.sda2.in‚0.03125 disk.drbd2041.out‚0 disk.dm-0.in‚ 87.8984375󴾂(:„ ™ÎŠÓóÖÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.4' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.8 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fù Úž¿‚(:“ ¨ÎŠÓ¶ŒÎ³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚295.033' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.34433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷C_=À‚(:²  ¨ÎŠÓ¼Óí³ disk.drbd1552.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 281.24609375 disk.drbd2118.out‚0# memory_percent‚92.0633316040039 network.recv‚655308) disk./.usagePercent‚70.51374713205674 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.sda.out‚ 281.24609375# disk.nvme0n1.out‚33345.84765625 disk.sda1.in‚0* proc.memory.percent‚11.835050582885742 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0% disk./.usageGB‚1260.8574142456055 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.dm-0.out‚33345.84765625 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.rbd1.out‚ 56.5185546875 disk.drbd1938.in‚0' proc.memory.availableMB‚ 1950.515625# proc.memory.rssMB‚ 2908.58203125 disk.drbd1285.out‚0 network.sent‚ 1.0760631e+07 disk.drbd2111.in‚0 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 cpu‚25.32383620722865t<íŸÁ‚(:” ·ÎŠÓÐ÷á³ gpu.0.gpu‚77.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.782' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.26066666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ô1>‚(:³  ·ÎŠÓ“ÿþ³ disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0# disk.dm-0.out‚33346.35498046875 disk.drbd1226.in‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8574714660645 disk.sda2.out‚ 281.53125 disk.drbd2118.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 87.8984375 cpu‚25.323780473409567 proc.cpu.threads‚17 network.recv‚655374 disk.drbd1356.out‚0 disk.drbd1226.out‚0# memory_percent‚92.0633316040039) disk./.usagePercent‚70.51375033212435 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 56.6201171875 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.835050582885742 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 87.8984375& disk.nvme0n1.out‚33346.35498046875 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 281.53125 disk.sda2.in‚0.03125 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 network.sent‚ 1.0760697e+07 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1950.515625 disk.drbd1817.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2908.58203125ÃZgžÃ‚(:“ ÆΊÓàýª³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.193' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.06433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^„\¢@Ä‚(:µ  ÆΊÓ‡˜Î³ network.recv‚655482# memory_percent‚92.0633316040039 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 281.984375 disk.sda.out‚ 281.984375 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 87.8984375 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚33346.53466796875 disk.drbd1510.out‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.835050582885742 disk.nvme0n1.in‚ 87.8984375 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2908.58203125 network.sent‚ 1.0760805e+07 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.rbd1.out‚ 56.6826171875# disk.dm-0.out‚33346.53466796875 disk.drbd1965.out‚0% disk./.usageGB‚1260.8572998046875 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0' proc.memory.availableMB‚ 1950.515625 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51374073192152 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 cpu‚25.323682077929327 disk.drbd1356.out‚0 disk.drbd2041.in‚0f”K‹4Å/ _wandbruntime‚11020  _runtime‚11020Áf—ÇÆÁ & lr-AdamW/pg1‚7.304317096293555e-05 & lr-AdamW/pg2‚7.304317096293555e-05 trainer/global_step‚6049 % _timestamp‚1.7848502575770369e+09 _runtime‚11020.93426736 _step‚501‘ˆoôÌÇÆ & lr-AdamW/pg1‚7.304317096293555e-05 & lr-AdamW/pg2‚7.304317096293555e-05 trainer/global_step‚6049 % _timestamp‚1.7848502575770369e+09 _runtime‚11020.93426736 _step‚501õæ&.84È/ _wandbruntime‚11020  _runtime‚11020QÏwHQ ÉP VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1747574843948032 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56160357 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52502836 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56160357 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1746878692274104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16121435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16121440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6311367263232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56170421 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚484388416220672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1741263502858240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6312712699392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚6049 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6311367263232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1741263502858240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1746877322234850 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34540925 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1747574843948032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1740576523241424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚502 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34541274 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1740576548669392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6300798993426 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600[¯ŠÑ (ʆP YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34540925 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1747574843948032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1740576523241424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚502 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34541274 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1740576548669392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6300798993426 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1747574843948032 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56160357 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52502836 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56160357 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1746878692274104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16121435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16121440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6311367263232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56170421 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚484388416220672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1741263502858240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6312712699392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚6049 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6311367263232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1741263502858240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1746877322234850 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976öCTÖ·4Ë/ _wandbruntime‚11022  _runtime‚11022îØŒ'Ì‹N $ _timestamp‚1.784850259054467e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52511203 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1747854775746560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚503 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52511197 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚11022.412065952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34546478 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1740855325015200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1747856147363328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7721611522560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚6049 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1741542412148736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6312389771264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18422403 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1740855350443168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6313735214592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108690720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1741542385975296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56169439 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108680636 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52511197 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1741542412148736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18422750 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6303164406464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1741542385975296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6313735214592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34546830 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56169439 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6301819795066 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824ÝKçÖÍN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1747854775746560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚503 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52511197 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚11022.412065952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34546478 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1740855325015200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1747856147363328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7721611522560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚6049 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1741542412148736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6312389771264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18422403 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1740855350443168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6313735214592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108690720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1741542385975296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56169439 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108680636 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52511197 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1741542412148736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18422750 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6303164406464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1741542385975296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6313735214592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34546830 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56169439 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6301819795066 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 $ _timestamp‚1.784850259054467e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52511203 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0÷eñã4Î/ _wandbruntime‚11022  _runtime‚11022¦Ý± Ï„ 1loss/SRqqt2/detection_loss‚0.6431951302825177 .loss/FBt2/detection_loss‚4.580639518121865 epoch‚21 _runtime‚11022.446854335 0loss/FRt2/assignment_loss‚0.8157060605356063 3loss/SRqqt2/assignment_loss‚0.45978244956298436 /loss/FRt2/detection_loss‚1.5226764202042986 1loss/SRqqt1/detection_loss‚0.9497815974277712 /loss/FBt1/detection_loss‚0.6334370576046608 trainer/global_step‚6049 $ _timestamp‚1.784850259089626e+09 0loss/FRt1/assignment_loss‚1.7235550313096974 2loss/FBt1/assignment_loss‚0.056474067311581444 1loss/FBt2/assignment_loss‚0.06126103239988441 /loss/FRt1/detection_loss‚0.5453459264017844 &loss/total_loss‚12.521551891398627 _step‚504 2loss/SRqqt1/assignment_loss‚0.5296976002359755u[¼)Љ 1loss/SRqqt2/detection_loss‚0.6431951302825177 .loss/FBt2/detection_loss‚4.580639518121865 epoch‚21 _runtime‚11022.446854335 0loss/FRt2/assignment_loss‚0.8157060605356063 3loss/SRqqt2/assignment_loss‚0.45978244956298436 /loss/FRt2/detection_loss‚1.5226764202042986 1loss/SRqqt1/detection_loss‚0.9497815974277712 /loss/FBt1/detection_loss‚0.6334370576046608 trainer/global_step‚6049 $ _timestamp‚1.784850259089626e+09 0loss/FRt1/assignment_loss‚1.7235550313096974 2loss/FBt1/assignment_loss‚0.056474067311581444 1loss/FBt2/assignment_loss‚0.06126103239988441 /loss/FRt1/detection_loss‚0.5453459264017844 &loss/total_loss‚12.521551891398627 _step‚504 2loss/SRqqt1/assignment_loss‚0.5296976002359755øMˆzŸÑ‚(:” ÕΊÓÓܶ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.668' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.88933333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K|üOGÒ‚(:¼  ÕΊÓèÙÞ³) disk./.usagePercent‚70.51374286529992 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 282.56640625 disk.sda2.in‚0.03125# disk.dm-0.out‚33347.00830078125 disk.sda.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 network.recv‚661301 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 282.56640625 disk.dm-0.in‚ 87.8984375& disk.nvme0n1.out‚33347.00830078125 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1950.16015625# proc.memory.rssMB‚ 2908.58203125 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.rbd1.out‚ 56.7685546875$ memory_percent‚92.06477801005045 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8573379516602 disk.drbd1673.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.835050582885742 cpu‚25.323626742160155 disk.drbd2118.out‚0 network.sent‚ 1.0829617e+07 disk.nvme0n1.in‚ 87.8984375 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1965.in‚0ΔCê“Ó‚(:ˆ äΊÓÒÖɳ gpu.0.gpu‚67.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.519' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.173 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V"H§EÔ‚(:º  äΊÓÎÈë³ disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0) proc.memory.availableMB‚ 1950.16796875 network.recv‚662117 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 cpu‚25.323526530625035 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 283.06640625 disk.drbd1289.in‚0 disk.sda1.in‚0* proc.memory.percent‚11.835050582885742% disk./.usageGB‚1260.8572807312012 disk.sda2.in‚0.03125 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 87.8984375" disk.dm-0.out‚33347.3857421875 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.sda.out‚ 283.06640625 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.06474622090658 disk.drbd1938.out‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51373966523231 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚33347.3857421875 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 57.0107421875 disk.drbd1817.in‚0 disk.sda.in‚0.03125 network.sent‚ 1.0830598e+07 disk.drbd1226.out‚0 disk.nvme0n1.in‚ 87.8984375# proc.memory.rssMB‚ 2908.58203125 proc.cpu.threads‚17ÿnY‘Õ‚(:† óΊÓÇØʳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.56' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.52 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆOËOGÖ‚(:¼  óΊÓͲë³ disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚33348.28857421875 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda.in‚0.03125) disk./.usagePercent‚70.51374073192152 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 network.sent‚ 1.0832924e+07 disk.sda2.out‚ 283.33984375 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 87.8984375) proc.memory.availableMB‚ 1949.91796875 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 network.recv‚662357 disk.drbd1356.in‚0& disk.nvme0n1.out‚33348.28857421875 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 57.1044921875 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0$ memory_percent‚92.06576347351074 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2908.58203125 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 283.33984375 cpu‚25.323449615753574* proc.memory.percent‚11.835050582885742 disk.dm-0.in‚ 87.8984375 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8572998046875 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚08‹-]Ÿ×‚(:” ‚ÏŠÓΫѳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.035' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.01166666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FZ]ýBØ‚(:·  ‚ÏŠÓŽÌì³) disk./.usagePercent‚70.51374307863776 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1550.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1949.9140625% disk./.usageGB‚1260.8573417663574 disk.dm-0.in‚ 87.8984375 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2908.58203125 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 57.1669921875 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0& disk.nvme0n1.out‚33348.92724609375 disk.drbd1510.in‚0 disk.sda.out‚ 283.7265625 disk.drbd1965.in‚0 disk.drbd1965.out‚0 cpu‚25.32339538506249 network.sent‚ 1.083869e+07 disk.drbd1931.out‚0 disk.drbd1552.in‚0$ memory_percent‚92.06577936808267 network.recv‚664261 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 283.7265625# disk.dm-0.out‚33348.92724609375 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.835050582885742/žÙ‚(:“ ‘ÏŠÓÚÚس gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.451' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.15033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤Ð—ÂFÚ‚(:»  ‘ÏŠÓÛëø³ disk.dm-0.in‚ 87.8984375 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0# disk.dm-0.out‚33349.83251953125 disk.drbd2078.out‚0 disk.rbd1.out‚ 57.2529296875 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 284.33984375 network.recv‚664393 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.sda1.out‚0 cpu‚25.323295897773747 proc.cpu.threads‚17 network.sent‚ 1.0839495e+07% disk./.usageGB‚1260.8573036193848 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚33349.83251953125 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0( proc.memory.availableMB‚ 1949.9140625 disk.drbd2111.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.06577936808267* proc.memory.percent‚11.835050582885742 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125) disk./.usagePercent‚70.51374094525936 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2908.58203125 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1817.out‚0 disk.sda2.out‚ 284.33984375sU|•Û‚(:’  ÏŠÓÎÀ²³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚281.84' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.94666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñpÎZFÜ‚(:»   ÏŠÓÉŠÒ³ disk.drbd2041.in‚0 disk.sda.in‚0.03125 network.recv‚665502) disk./.usagePercent‚70.51373987857015 disk.dm-0.in‚ 87.8984375 disk.sda.out‚ 284.64453125 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 1949.91015625 disk.drbd2078.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚33350.00048828125 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚33350.00048828125 disk.sda1.out‚0 disk.drbd1817.out‚0 network.sent‚ 1.0844611e+07 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0$ memory_percent‚92.06579526265463 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda2.in‚0.03125 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 284.64453125 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2908.58203125% disk./.usageGB‚1260.8572845458984 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 cpu‚25.32322105428308 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.835050582885742 disk.drbd1828.out‚0 disk.rbd1.out‚ 57.3154296875 disk.drbd1550.out‚0Z.©4Ý/ _wandbruntime‚11107  _runtime‚11107ÙÔ)ÈÞ & lr-AdamW/pg1‚7.239254772345282e-05 & lr-AdamW/pg2‚7.239254772345282e-05 trainer/global_step‚6099 % _timestamp‚1.7848503441101794e+09 _runtime‚11107.467409015 _step‚505ö#ñÍßÇ % _timestamp‚1.7848503441101794e+09 _runtime‚11107.467409015 _step‚505 & lr-AdamW/pg1‚7.239254772345282e-05 & lr-AdamW/pg2‚7.239254772345282e-05 trainer/global_step‚6099ùמЅ4à/ _wandbruntime‚11107  _runtime‚11107Lçs×(á‚P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1761571498029056 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1755208967383040 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚506 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6362556819456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1761571498029056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6363902255616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 _runtime‚11107.468803266 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16253435 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1754516637358192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6351902876234 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109547559 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1755208941209600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34820025 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1755208941209600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56616309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚480597385299968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782978946560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16253440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109537489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚488380419707904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚488379996155392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52921186 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34819678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 trainer/global_step‚6099 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1761572869638656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56616309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6362556819456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56626373 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6353247487520 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52921180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚480597017208832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7783034407936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18566243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 % _timestamp‚1.7848503441112182e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6353247487520 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52921180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚480597017208832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7783034407936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18566243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 % _timestamp‚1.7848503441112182e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1761571498029056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6363902255616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 _runtime‚11107.468803266 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16253435 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1754516637358192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6351902876234 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109547559 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1755208941209600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34820025 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1755208941209600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56616309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚480597385299968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782978946560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16253440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109537489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚488380419707904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚488379996155392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52921186 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34819678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 trainer/global_step‚6099 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1761572869638656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56616309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6362556819456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005úœHÜ4ã/ _wandbruntime‚11108  _runtime‚11108-Gªü'äŒN =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109554938 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6364924770816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1754795439131968 _step‚507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚488459836897792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6352923677874 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6363579327488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34825237 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52929553 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56625391 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1755487876673536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1761148337381874 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109565022 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109554938 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚6099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848503455877824e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1755487876673536 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1761852801444352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52929547 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚488460254151680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18569162 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56635469 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56625391 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7784282485760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52929553 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16256080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109565022 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6364924770816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1761852801444352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1755487850500096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18569499 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚337 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109565022 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109554938 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚6099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848503455877824e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1755487876673536 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1761852801444352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52929547 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚488460254151680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18569162 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56635469 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56625391 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7784282485760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52929553 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16256080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109565022 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6364924770816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1761852801444352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1755487850500096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18569499 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚337 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34825237 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52929553ûÁ/oˆ4æ/ _wandbruntime‚11108  _runtime‚11108]äøÕ çƒ 2loss/SRqqt2/assignment_loss‚0.5347177054647538 3loss/FBt1/assignment_loss‚0.0009814984331495834 /loss/FRt1/detection_loss‚0.5749160058924585 /loss/FRt2/detection_loss‚1.4322153711511991 trainer/global_step‚6099 2loss/FBt2/assignment_loss‚0.008700638965810551 1loss/SRqqt1/detection_loss‚0.8473802793092268 .loss/FBt2/detection_loss‚0.719129898701236 2loss/SRqqt1/assignment_loss‚0.5063915119803376 0loss/SRqqt2/detection_loss‚0.602749949812038 /loss/FBt1/detection_loss‚0.5289038382470608 $ _timestamp‚1.784850345623985e+09 _step‚508 0loss/FRt1/assignment_loss‚1.6702392180403787 0loss/FRt2/assignment_loss‚0.6991896148171284 %loss/total_loss‚8.125515530814777 epoch‚21 _runtime‚11108.981192803Äd`çèˆ 2loss/SRqqt1/assignment_loss‚0.5063915119803376 0loss/SRqqt2/detection_loss‚0.602749949812038 /loss/FBt1/detection_loss‚0.5289038382470608 $ _timestamp‚1.784850345623985e+09 _step‚508 0loss/FRt1/assignment_loss‚1.6702392180403787 0loss/FRt2/assignment_loss‚0.6991896148171284 %loss/total_loss‚8.125515530814777 epoch‚21 _runtime‚11108.981192803 2loss/SRqqt2/assignment_loss‚0.5347177054647538 3loss/FBt1/assignment_loss‚0.0009814984331495834 /loss/FRt1/detection_loss‚0.5749160058924585 /loss/FRt2/detection_loss‚1.4322153711511991 trainer/global_step‚6099 2loss/FBt2/assignment_loss‚0.008700638965810551 1loss/SRqqt1/detection_loss‚0.8473802793092268 .loss/FBt2/detection_loss‚0.719129898701236üVV±'’é‚(:‡ ¯ÏŠÓà¯Á³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚307.62' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.54 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_Y\û=ê‚(:²  ¯ÏŠÓð™ã³! disk.dm-0.out‚33350.416015625 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0( disk./.usagePercent‚70.5140159377362 disk.drbd1289.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1950.09375 disk.sda.out‚ 285.1015625 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 cpu‚25.323166741442932 network.recv‚665568 disk.dm-0.in‚ 87.8984375 disk.drbd2041.out‚0 network.sent‚ 1.0844677e+07 disk.drbd1584.in‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8622207641602 disk.drbd1817.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 57.5966796875 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2908.58203125 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.835050582885742 proc.cpu.threads‚17 disk.drbd1289.out‚0$ disk.nvme0n1.out‚33350.416015625 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 285.1015625 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1552.out‚0$ memory_percent‚92.06504821777344 disk.drbd1584.out‚0¯MÇ­žë‚(:“ ¾ÏŠÓóÈÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚276.878' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.29266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å…ôG?ì‚(:´  ¾ÏŠÓ«äõ³ disk.rbd1.out‚ 57.6591796875 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0# memory_percent‚92.0650323232015) proc.memory.availableMB‚ 1950.09765625* proc.memory.percent‚11.835050582885742 proc.cpu.threads‚17) disk./.usagePercent‚70.51406159203415 disk.drbd1510.out‚0 disk.drbd1289.out‚0 cpu‚25.323090134467936 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda2.in‚0.03125 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2908.58203125 disk.sda.in‚0.03125 disk.sda.out‚ 285.5234375 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0$ disk.nvme0n1.out‚33350.544921875 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 285.5234375 network.recv‚665634 disk.nvme0n1.in‚ 87.8984375 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0! disk.dm-0.out‚33350.544921875 disk.drbd1828.in‚0 disk.drbd1828.out‚0 network.sent‚ 1.0844743e+07 disk.dm-0.in‚ 87.8984375 disk.sda1.out‚0$ disk./.usageGB‚1260.863037109375 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0Ò‘¼Bí‚(:’ ÍÏŠÓŒ›Ô³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.93' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.64333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿7¾=î‚(:¶  ÍÏŠÓ‘—ó³) disk./.usagePercent‚70.5140541ú^Ÿ2520973 disk.drbd1673.out‚0# disk.dm-0.out‚33351.71826171875 disk.drbd1356.out‚0% disk./.usageGB‚1260.8629035949707 disk.sda2.out‚ 285.90625 disk.drbd2041.in‚0 disk.drbd1828.in‚0 cpu‚25.323014527998627 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1951.26953125 network.recv‚670025 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1817.in‚0$ memory_percent‚92.06026395161948 network.sent‚ 1.0910417e+07 disk.nvme0n1.in‚ 87.8984375 disk.drbd1817.out‚0 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2908.58203125& disk.nvme0n1.out‚33351.71826171875 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 57.6591796875 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.835050582885742 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 285.90625 disk.drbd1931.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.dm-0.in‚ 87.8984375 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0%“Œ“ï‚(:ˆ ÜÏŠÓÁâÓ³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.918' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ž}ùCð‚(:¸  ÜÏŠÓîñí³ network.sent‚ 1.0911395e+07 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.06039110819498* proc.memory.percent‚11.835050582885742 disk.drbd1828.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2908.58203125 disk.drbd1817.out‚0 disk.sda.out‚ 286.18359375 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0( disk./.usagePercent‚70.5140487917637$ disk./.usageGB‚1260.862808227539 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.rbd1.out‚ 57.69140625 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125) proc.memory.availableMB‚ 1951.23828125 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚33352.12060546875 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 87.8984375 cpu‚25.322983884927115 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.sda2.out‚ 286.18359375 disk.drbd1938.in‚0# disk.dm-0.out‚33352.12060546875 network.recv‚670941rP žñ‚(:“ ëÏŠÓ™ŸË³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.03' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.343333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 7›ñ4ò‚(:©  ëÏŠÓ¬Øì³ disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.dm-0.out‚ 33353.171875 disk.drbd2041.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 87.8984375# proc.memory.rssMB‚ 2908.58203125 disk.drbd1285.in‚0 disk.sda1.in‚0$ memory_percent‚92.05942153930664 cpu‚25.322975475899888 disk.drbd1965.out‚0 disk.drbd1550.in‚0( proc.memory.availableMB‚ 1951.4765625* proc.memory.percent‚11.835050582885742 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1356.out‚0 network.recv‚671863 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 network.sent‚ 1.0914374e+07 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.rbd1.out‚ 57.79296875 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sda2.out‚ 286.59375 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51404985845289 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8628273010254! disk.nvme0n1.out‚ 33353.171875 disk.sda1.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd2118.in‚0 disk.sda.out‚ 286.59375˜SÐÇ;ó‚(:°  úÏŠÓׯϳ disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.sda.out‚ 287.0546875 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0# disk.nvme0n1.out‚33353.30859375 disk.drbd1289.out‚0) disk./.usagePercent‚70.51404985845289 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 287.0546875 disk.dm-0.in‚ 87.8984375 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 57.85546875$ memory_percent‚92.06040700276694 disk.dm-0.out‚33353.30859375 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.drbd1226.out‚0* proc.memory.percent‚11.835050582885742 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2908.58203125' proc.memory.availableMB‚ 1951.234375% disk./.usageGB‚1260.8628273010254 disk.drbd1931.in‚0 disk.drbd1828.in‚0 network.sent‚ 1.0919364e+07 network.recv‚673035 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 cpu‚25.322856747129077 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 87.8984375 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0î#õÚŸô‚(:” úÏŠÓ¼¿²³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.663' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.55433333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡„½Y4õ/ _wandbruntime‚11194  _runtime‚11194µD(3Äö¾ trainer/global_step‚6149 # _timestamp‚1.78485043066613e+09 _runtime‚11194.023350259 _step‚509 % lr-AdamW/pg1‚7.17401119452398e-05 % lr-AdamW/pg2‚7.17401119452398e-05óhûÉ÷à % lr-AdamW/pg1‚7.17401119452398e-05 % lr-AdamW/pg2‚7.17401119452398e-05 trainer/global_step‚6149 # _timestamp‚1.78485043066613e+09 _runtime‚11194.023350259 _step‚509ý!6Øê4ø/ _wandbruntime‚11194  _runtime‚11194ÉòêÄ(ù‚P trainer/global_step‚6149 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7845834995712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484526335506432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110421861 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1775568152110080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53339530 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6415091811840 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1769154405734400 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1769154431907840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18713650 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6403006759042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1768456700619024 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6413746375680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1774861077417320 % _timestamp‚1.7848504306671584e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492372125526528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1769154405734400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53339536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚484526703597568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864Ý\ (ú‡P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1768456700619024 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6413746375680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1774861077417320 % _timestamp‚1.7848504306671584e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492372125526528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1769154405734400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53339536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚484526703597568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚6149 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7845834995712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484526335506432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110421861 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1775568152110080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53339530 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6415091811840 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1769154405734400 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1769154431907840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚354 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18713650 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6403006759042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0þÝ—7‡4û/ _wandbruntime‚11195  _runtime‚11195îÎsKÓüŒN REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57091421 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7847045361152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35104663 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57081343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1768735502392800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53347903 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚484604921872384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110439324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1768735527820768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6404027560682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110439324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53347897 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492452382390272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚484605289963520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1775849455525376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16388080 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1769433315024896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53347903 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110429240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6414768883712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7847092426752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328Ú÷Wg'ý‘N _runtime‚11195.502180871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18716588 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1775848083908608 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110439324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53347897 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492452382390272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚484605289963520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1775849455525376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16388080 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1769433315024896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53347903 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110429240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6414768883712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7847092426752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57091421 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7847045361152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35104663 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57081343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1768735502392800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53347903 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚484604921872384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110439324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1768735527820768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6404027560682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F‚‚(:³  ‰ÐŠÓƒ¢ì³ disk.drbd1550.in‚0 network.sent‚ 1.092239e+07 disk.sda.in‚0.03125 disk.drbd1289.in‚0 network.recv‚673948 disk.drbd2118.in‚0 disk.dm-0.in‚ 87.8984375 disk.drbd1965.in‚0 disk.drbd1931.in‚0" disk.dm-0.out‚33353.8271484375 disk.drbd2078.in‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚33353.8271484375 disk.drbd1226.in‚0% disk./.usageGB‚1260.8582000732422 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0$ memory_percent‚91.74248377482095 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51379107965195 disk.rbd1.out‚ 57.85546875 disk.drbd1510.out‚0 disk.sda2.out‚ 287.6953125 cpu‚25.322847430919378 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.sda.out‚ 287.6953125 disk.drbd1289.out‚0 disk.drbd1938.in‚0( proc.memory.availableMB‚ 2029.3671875 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.387602488199871 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 87.8984375 disk.sda1.in‚0" proc.memory.rssMB‚ 2798.6171875t}˜ÿžƒ‚(:“ ˜ÐŠÓ°ÑƳ gpu.0.gpu‚70.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚106.511' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.50366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o®wµ@„‚(:µ  ˜ÐŠÓÉ‘æ³ network.recv‚675963% disk./.usageGB‚1260.8581466674805 disk.drbd1931.out‚0 disk.rbd1.out‚ 57.8876953125$ memory_percent‚92.08733240763345 disk.sda1.out‚0 disk.sda.out‚ 288.0234375 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 88.5703125% disk.nvme0n1.out‚33354.0419921875 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0" disk.dm-0.out‚33354.0419921875 disk.drbd2078.in‚0) disk./.usagePercent‚70.51378809292218 disk.sda2.out‚ 288.0234375 disk.drbd1828.out‚0( proc.memory.availableMB‚ 1944.6171875! proc.memory.rssMB‚ 2883.109375 network.sent‚ 1.0980193e+07 disk.drbd2041.out‚0 disk.drbd1510.in‚0 cpu‚25.324283397667855 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.731402079264324 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 88.5703125 disk.drbd2111.out‚0 disk.rbd1.in‚0ˆà4…/ _wandbruntime‚11232  _runtime‚11232=xÃ(+†úu 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6801127791404724 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3590221107006073 *jet/accuracy_1_of_3‚0.9307618737220764 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7626155018806458 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4859614670276642 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 +jet/accuracy_3_of_3‚0.07215265184640884 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3905916213989258 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38982754945755005 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8686302900314331 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6834936738014221 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 (particle/accuracy‚0.8524401783943176 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9133680462837219 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14592130482196808 _runtime‚11232.251229784 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21440443396568298 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07307741791009903 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26285356283187866 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚8.145188942421802 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.310886949300766 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8030160665512085 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.272409051656723 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1534949094057083 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1716090887784958 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762651085853577 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.258760541677475 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6618232727050781 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3112821578979492 *jet/accuracy_1_of_2‚0.7888635993003845 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7848504688931048e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2726998031139374 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7924241423606873 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7283366918563843 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.39546582102775574 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8328462839126587 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9393061399459839 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8328462839126587 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890350103378296 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24761192500591278 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8311303853988647 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 _step‚513 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.054496586322784424 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018720822408795357 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.737433135509491 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6891242265701294 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6166666746139526 /particle/accuracy_4_of_4‚0.9792338013648987 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974337220191956 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5187720060348511 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04693001136183739 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7207332253456116 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3108298182487488 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.803365170955658 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4627349078655243 *jet/accuracy_2_of_2‚0.1664564311504364 +jet/accuracy_3_of_4‚0.11857572942972183 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46454352140426636 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 +particle/sensitivity‚0.6497215628623962 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6545692086219788 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6916803121566772 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3108298182487488 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03311634808778763 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7245049476623535 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02569630555808544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7014561295509338 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.739862859249115 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6141628623008728 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7170335650444031 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5051282048225403 +particle/specificity‚0.9078128933906555 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8323360681533813 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24405673146247864 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 trainer/global_step‚6159 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6226862072944641 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4358974099159241 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7528261542320251 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3331814408302307 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.64225834608078 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23765014111995697 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32261961698532104 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762651085853577 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36443519592285156 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42296653985977173 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6106502413749695 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6545692086219788 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4953966736793518 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03077775426208973 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7320512533187866 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02735874243080616 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8665145635604858 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.811291515827179 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11648042500019073 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5811538696289062 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6468577980995178 5 Purity/1gf ØFRt0SRqqt*FBt/FRt_purity‚0.32315793633461 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05766809731721878 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3663341701030731 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5556835532188416 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8322150111198425 *jet/accuracy_2_of_3‚0.5964409112930298 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7193896174430847 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5917451977729797 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7216351628303528 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6494023203849792 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018720822408795357 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333730697632 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6971720457077026 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7573398947715759 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9743425250053406 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9921409487724304 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6810866594314575 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚21 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2608093023300171 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.8346272110939026 'particle/f_score‚0.6538980603218079 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27161842584609985 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6564298868179321 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6389457583427429 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3402542173862457 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21106933057308197 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964097142219543 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14057736098766327 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7979426980018616 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15861760079860687 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6866771578788757 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6993218660354614 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2809523940086365 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26070457696914673 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7225755453109741 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333730697632 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2420504242181778 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3245669901371002 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30274224281311035ñ2/];‡ÿu 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7320512533187866 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02735874243080616 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8665145635604858 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.811291515827179 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11648042500019073 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5811538696289062 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6468577980995178 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32315793633461 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05766809731721878 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3663341701030731 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5556835532188416 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8322150111198425 *jet/accuracy_2_of_3‚0.5964409112930298 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7193896174430847 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5917451977729797 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7216351628303528 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6494023203849792 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018720822408795357 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333730697632 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6971720457077026 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7573398947715759 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9743425250053406 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9921409487724304 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6810866594314575 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚21 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2608093023300171 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.8346272110939026 'particle/f_score‚0.6538980603218079 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27161842584609985 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6564298868179321 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6389457583427429 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3402542173862457 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21106933057308197 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964097142219543 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14057736098766327 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7979426980018616 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15861760079860687 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6866771578788757 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6993218660354614 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2809523940086365 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26070457696914673 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7225755453109741 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333730697632 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2420504242181778 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3245669901371002 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30274224281311035 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6801127791404724 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3590221107006073 *jet/accuracy_1_of_3‚0.9307618737220764 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7626155018806458 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4859614670276642 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 +jet/accuracy_3_of_3‚0.07215265184640884 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3905916213989258 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38982754945755005 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8686302900314331 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6834936738014221 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 (particle/accuracy‚0.8524401783943176 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9133680462837219 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14592130482196808 _runtime‚11232.251229784 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21440443396568298 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07307741791009903 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26285356283187866 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚8.145188942421802 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.310886949300766 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8030160665512085 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.272409051656723 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1534949094057083 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1716090887784958 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762651085853577 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.258760541677475 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6618232727050781 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3112821578979492 *jet/accuracy_1_of_2‚0.7888635993003845 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7848504688931048e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2726998031139374 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7924241423606873 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7283366918563843 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.39546582102775574 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8328462839126587 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9393061399459839 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8328462839126587 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890350103378296 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24761192500591278 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8311303853988647 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 _step‚513 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.054496586322784424 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018720822408795357 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.737433135509491 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6891242265701294 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6166666746139526 /particle/accuracy_4_of_4‚0.9792338013648987 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974337220191956 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5187720060348511 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04693001136183739 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7207332253456116 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3108298182487488 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.803365170955658 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4627349078655243 *jet/accuracy_2_of_2‚0.1664564311504364 +jet/accuracy_3_of_4‚0.11857572942972183 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46454352140426636 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 +particle/sensitivity‚0.6497215628623962 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6545692086219788 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6916803121566772 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3108298182487488 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03311634808778763 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7245049476623535 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02569630555808544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7014561295509338 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.739862859249115 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6141628623008728 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7170335650444031 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5051282048225403 +particle/specificity‚0.9078128933906555 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8323360681533813 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24405673146247864 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 trainer/global_step‚6159 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6226862072944641 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4358974099159241 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7528261542320251 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3331814408302307 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.64225834608078 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23765014111995697 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32261961698532104 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762651085853577 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36443519592285156 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42296653985977173 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6106502413749695 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6545692086219788 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4953966736793518 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03077775426208973 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192"ò¯Aˆ‚(:¶  §ÐŠÓÿ‚´ disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0) proc.memory.availableMB‚ 2913.31640625$ disk./.usageGB‚1260.862003326416 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 88.5703125 disk.drbd1356.out‚0$ disk.nvme0n1.out‚33354.388671875 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1828.in‚0$ memory_percent‚88.14568519592285 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 88.5703125 network.sent‚ 1.0980259e+07) disk./.usagePercent‚70.51400377747929 disk.drbd1817.out‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.746835708618164 cpu‚25.32691230625408 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2886.90234375 network.recv‚676029 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.sda2.out‚ 288.25390625 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda.out‚ 288.25390625! disk.dm-0.out‚33354.388671875 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.rbd1.out‚ 57.9892578125 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0÷Q•œ‰‚(:‘ §ÐŠÓÿÞ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚57.0 gpu.0.powerWatts‚105.596' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.19866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—3·øžŠ‚(:“ ¶ÐŠÓÉƳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚106.052' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.35066666666667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æëk;‹‚(:°  ¶ÐŠÓƒÚè³ disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.sda2.out‚ 288.6015625 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 58.0517578125 disk.drbd1510.out‚0 disk.sda.out‚ 288.6015625 disk.drbd1226.in‚0* proc.memory.percent‚11.746978759765625 disk.drbd2041.out‚0 disk.dm-0.out‚34324.23046875 disk.drbd1817.out‚0( disk./.usagePercent‚70.5140069775469 disk.drbd1289.in‚0 disk.drbd1965.out‚0# disk.nvme0n1.out‚34324.23046875$ memory_percent‚88.14765612284342 cpu‚25.325747488529963 disk.drbd2078.out‚0 disk.sda.in‚0.03125$ disk./.usageGB‚1260.862060546875 disk.drbd1938.in‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 network.sent‚ 1.1039732e+07 network.recv‚680405 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 2912.83203125 proc.cpu.threads‚17»&,ŽžŒ‚(:“ ÅЊÓì¢Ä³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚296.008' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.66933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0И ”=‚(:²  ÅЊÓæ¶ä³ disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8602561950684 disk.nvme0n1.in‚ 88.57421875 disk.dm-0.in‚ 88.57421875 cpu‚25.325647093156974 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 288.984375 disk.drbd1673.out‚0 disk.drbd1938.in‚0$ disk.nvme0n1.out‚34324.806640625 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.746978759765625 network.sent‚ 1.1040708e+07! disk.dm-0.out‚34324.806640625 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0) proc.memory.availableMB‚ 2912.80078125 disk.drbd1289.out‚0 disk.rbd1.out‚ 58.0517578125 disk.drbd1673.in‚0 disk.sda2.out‚ 288.984375 disk.drbd1510.out‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0$ memory_percent‚88.14778327941895 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 network.recv‚681347 disk.drbd1931.in‚0) disk./.usagePercent‚70.51390606874817 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0¢þIŸŽ‚(:” ÔЊÓŦå³ gpu.0.gpu‚71.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.353' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.11766666666666 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=·z®@‚(:µ  ÔЊÓă´ disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 58.083984375 disk.drbd1965.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2118.out‚0$ memory_percent‚88.14781506856283 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1550.out‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 289.32421875% disk./.usageGB‚1260.8564186096191 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.sda.out‚ 289.32421875 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd1673.in‚0) proc.memory.availableMB‚ 2912.79296875 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚35294.544921875 disk.sda2.in‚0.03125 proc.memory.rssMB‚ 2886.9375 disk.drbd1226.out‚0 disk.sda.in‚0.03125 cpu‚25.325589879574743 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.746978759765625 network.sent‚ 1.1043693e+07 network.recv‚682379! disk.dm-0.out‚35294.544921875 disk.drbd1510.out‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51369145088027 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0ì^׈Ÿ‚(:” ãЊÓÏ º³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.582' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.193999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p¶J:‘‚(:¯  ãЊÓúس% disk./.usageGB‚1260.8564376831055 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0' proc.memory.availableMB‚ 2912.796875 disk.sda2.in‚0.03125" disk.dm-0.out‚35294.9736328125 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 58.185546875) disk./.usagePercent‚70.51369251756948 disk.drbd1828.out‚0 disk.drbd1226.in‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda2.out‚ 289.59375 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.drbd1584.in‚0 cpu‚25.325513460140705* proc.memory.percent‚11.746978759765625 disk.sda.out‚ 289.59375 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.sda1.out‚0 network.recv‚683348 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚35294.9736328125 disk.drbd1673.in‚0 disk.dm-0.in‚ 88.57421875 network.sent‚ 1.1046137e+07 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0$ memory_percent‚88.14779917399088Ò\½m4’/ _wandbruntime‚11305  _runtime‚113057Š ”Ç“Á & lr-AdamW/pg1‚7.108594914767894e-05 & lr-AdamW/pg2‚7.108594914767894e-05 trainer/global_step‚6199 $ _timestamp‚1.784850542071088e+09 _runtime‚11305.428330997 _step‚514e‚•{Ì”Æ $ _timestamp‚1.784850542071088e+09 _runtime‚11305.428330997 _step‚514 & lr-AdamW/pg1‚7.108594914767894e-05 & lr-AdamW/pg2‚7.108594914767894e-05 trainer/global_step‚6199‚茖v4•/ _wandbruntime‚11305  _runtime‚11305?PkŠh –‚P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6459719481164 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111422837 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚496420399797248 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1784955815886960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6461064092450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1785660573157376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35412086 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1784955790458992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1792131101355008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1785660546983936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_poolÚ; .allocated‚18888083 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57575525 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848505420722215e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1791416879979410 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1791415509940156 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6471899807232 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1785660546983936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111432907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53847312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚488504653317120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _runtime‚11305.429783075 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7915793552896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488505021408256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16524327 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7915746480128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _step‚515 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53847318 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57585589 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57575525 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35412415 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53847318 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1792132472964608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6470554371072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16524332 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6471899807232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57585589 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚324 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18887759 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1792132472964608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6470554371072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚496420814961152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0-“û (—‡P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488505021408256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16524327 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7915746480128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _step‚515 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53847318 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57585589 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57575525 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35412415 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53847318 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1792132472964608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6470554371072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16524332 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6471899807232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57585589 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚324 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18887759 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1792132472964608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6470554371072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚496420814961152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6459719481164 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111422837 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚496420399797248 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1784955815886960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6461064092450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1785660573157376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35412086 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1784955790458992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1792131101355008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1785660546983936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18888083 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57575525 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848505420722215e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1791416879979410 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1791415509940156 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6471899807232 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1785660546983936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111432907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53847312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚488504653317120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _runtime‚11305.429783075 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7915793552896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112ƒ1íä'4˜/ _wandbruntime‚11306  _runtime‚11306Ö†"'™‹N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚496500234072064 _runtime‚11306.90496577 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111450370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6472922322432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53855679 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111440286 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488583239683072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57594685 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1792412404770304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6462084894202 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111450370 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57584607 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111440286 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6471576879104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1791696702554938 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35417494 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6460740282804 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7916994388992 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1785234617660736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1785939456274432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7917045648896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1792411033153536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6471576879104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57594685 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18890527 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚496500653423104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1791695332515572 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35417818 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚324 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1785939482447872 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848505435473547e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35417494 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6460740282804 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7916994388992 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1785234617660736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1785939456274432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7917045648896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1792411033153536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6471576879104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57594685 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18890527 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚496500653423104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1791695332515572 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35417818 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚324 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1785939482447872 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848505435473547e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111450370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6472922322432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53855679 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111440286 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488583239683072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57594685 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1792412404770304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6462084894202„ÅÿÖ¢4›/ _wandbruntime‚11306  _runtime‚11306;÷|¨ œ† _step‚517 0loss/FRt2/assignment_loss‚0.7664076309876154 /loss/FRt1/detection_loss‚0.5293128386336342 /loss/FRt2/detection_loss‚1.4055563188533005 1loss/SRqqt2/detection_loss‚0.6299885423077015 trainer/global_step‚6199 epoch‚22 0loss/FRt1/assignment_loss‚1.7150143321707716 2loss/SRqqt1/assignment_loss‚0.6049267075988205 2loss/SRqqt2/assignment_loss‚0.4724907262703666 2loss/FBt2/assignment_loss‚0.019359033357287055 1loss/SRqqt1/detection_loss‚0.9482654778528269 /loss/FBt1/detection_loss‚0.9889325143449241 3loss/FBt1/assignment_loss‚0.0011638324310505599 /loss/FBt2/detection_loss‚0.3108329301373444 %loss/total_loss‚8.392250884945643 % _timestamp‚1.7848505435861242e+09 _runtime‚11306.943362497#õ¹ ‹ 3loss/FBt1/assignment_loss‚0.0011638324310505599 /loss/FBt2/detection_loss‚0.3108329301373444 %loss/total_loss‚8.392250884945643 % _timestamp‚1.7848505435861242e+09 _runtime‚11306.943362497 _step‚517 0loss/FRt2/assignment_loss‚0.7664076309876154 /loss/FRt1/detection_loss‚0.5293128386336342 /loss/FRt2/detection_loss‚1.4055563188533005 1loss/SRqqt2/detection_loss‚0.6299885423077015 trainer/global_step‚6199 epoch‚22 0loss/FRt1/assignment_loss‚1.7150143321707716 2loss/SRqqt1/assignment_loss‚0.6049267075988205 2loss/SRqqt2/assignment_loss‚0.4724907262703666 2loss/FBt2/assignment_loss‚0.019359033357287055 1loss/SRqqt1/detection_loss‚0.9482654778528269 /loss/FBt1/detection_loss‚0.9889325143449241…Í'¿µžž‚(:“ òЊÓÔ¶Ó³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.092' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.69733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†áÕ„=Ÿ‚(:²  òЊÓ·âë³ disk.drbd1550.out‚0 proc.memory.rssMB‚ 2886.9375" disk.dm-0.out‚35295.0908203125 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 network.recv‚683654 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd2078.in‚0 cpu‚25.32545692261141 disk.drbd1552.out‚0 disk.sda.out‚ 290.140625 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 290.140625 disk.rbd1.in‚0% disk.nvme0n1.out‚35295.0908203125* proc.memory.percent‚11.746978759765625 disk.sda2.in‚0.03125 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 network.sent‚ 1.1049542e+07 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 2912.73046875) disk./.usagePercent‚70.51369251756948 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 58.248046875 disk.drbd2118.out‚0 disk.drbd2078.out‚0$ memory_percent‚88.14806938171387 disk.drbd1828.out‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8564376831055 disk.sda.in‚0.03125 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0¹©Š¤Ÿ ‚(:” ÑŠÓ¥‰Ü³ gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.444' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.14800000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n&„D¡‚(:¹  ÑŠÓêßü³ disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0) proc.memory.availableMB‚ 2912.73046875) disk./.usagePercent‚70.51374777207026 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚35295.76025390625 disk.drbd1552.out‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 58.333984375 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0$ memory_percent‚88.14806938171387 disk.nvme0n1.in‚ 88.57421875 disk.drbd1584.out‚0 disk.drbd1938.out‚0# disk.dm-0.out‚35295.76025390625 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 88.57421875 disk.drbd1285.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 network.sent‚ 1.1111957e+07 network.recv‚687234% disk./.usageGB‚1260.8574256896973 disk.sda1.in‚0 cpu‚25.325422680761744 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 290.88671875 disk.sda2.out‚ 290.88671875* proc.memory.percent‚11.746978759765625 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0Òø‡)=¢‚(:²  ÑŠÓá²ò³ disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 cpu‚25.32532359328794 disk.drbd1817.out‚0 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 88.57421875* proc.memory.percent‚11.746978759765625 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1510.out‚0$ memory_percent‚88.14806938171387 disk.drbd2078.in‚0 disk.rbd1.out‚ 58.576171875 disk.drbd1938.out‚0) proc.memory.availableMB‚ 2912.73046875" disk.dm-0.out‚35296.2333984375 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 network.recv‚688143) disk./.usagePercent‚70.51374542535402 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 proc.memory.rssMB‚ 2886.9375% disk./.usageGB‚1260.8573837280273 disk.drbd1931.in‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 291.109375 disk.drbd2111.in‚0% disk.nvme0n1.out‚35296.2333984375 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 network.sent‚ 1.1112934e+07 disk.drbd1673.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.sda.out‚ 291.109375 disk.drbd1673.out‚0'™½d“£‚(:ˆ ÑŠÓœÚÔ³ gpu.0.gpu‚77.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.786' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤$ÜŸ¤‚(:” ŸÑŠÓ³¦°³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.624' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.87466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e»‘@¥‚(:µ  ŸÑŠÓÿÓγ$ memory_percent‚88.14806938171387 disk.dm-0.in‚ 88.57421875 disk.rbd1.out‚ 58.677734375 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8574028015137 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1289.in‚0& disk.nvme0n1.out‚35297.87841796875 disk.drbd1552.out‚0 disk.drbd1285.out‚0 proc.memory.rssMB‚ 2886.9375 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 291.421875* proc.memory.percent‚11.746978759765625 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1938.in‚0 network.sent‚ 1.1115386e+07 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 cpu‚25.325247635273154 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 2912.73046875 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.sda.out‚ 291.421875) disk./.usagePercent‚70.51374649204321 disk.drbd2078.out‚0# disk.dm-0.out‚35297.87841796875 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 network.recv‚689040 disk.drbd2078.in‚0 disk.drbd2111.in‚01pz”¦‚(:‰ ®ÑŠÓ¬•Ó³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚309.438' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚103.146 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®^å6?§‚(:´  ®ÑŠÓ¿Òó³ network.recv‚689412 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0( proc.memory.availableMB‚ 2912.9765625 disk.drbd1817.in‚0 disk.sda.in‚0.03125# disk.dm-0.out‚35298.12158203125 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 cpu‚25.325191184039575 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0$ memory_percent‚88.14706802368164* proc.memory.percent‚11.746978759765625 disk.drbd1965.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8574256896973 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1965.in‚0 disk.rbd1.out‚ 58.826171875 disk.nvme0n1.in‚ 88.57421875& disk.nvme0n1.out‚35298.12158203125 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51374777207026 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0 proc.memory.rssMB‚ 2886.9375 disk.sda2.in‚0.03125 disk.sda2.out‚ 292.046875 disk.drbd1584.in‚0 disk.drbd1817.out‚0 network.sent‚ 1.1120782e+07 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 292.046875 disk.drbd1931.in‚0 disk.drbd2078.in‚0”哨‚(:ˆ ½ÑŠÓò¾³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.735' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.245 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àž•<©‚(:±  ½ÑŠÓÊÈÞ³% disk./.usageGB‚1260.8574256896973 disk.drbd2041.out‚0* proc.memory.percent‚11.746978759765625 disk.drbd1226.out‚0 cpu‚25.325093154245746 network.recv‚691398 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 292.515625 disk.sda.out‚ 292.515625 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0$ memory_percent‚88.14584414164224 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1123994e+07 disk.dm-0.in‚ 88.57421875 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0) proc.memory.availableMB‚ 2913.27734375 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.rbd1.out‚ 58.826171875 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0$ disk.nvme0n1.out‚35298.673828125 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51374777207026 disk.rbd1.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0! disk.dm-0.out‚35298.673828125 disk.drbd2111.in‚0 proc.memory.rssMB‚ 2886.9375 disk.sda2.in‚0.03125p7É<4ª/ _wandbruntime‚11392  _runtime‚11392æ|uÈ«Â _runtime‚11392.217045001 _step‚518 & lr-AdamW/pg1‚7.043014507652537e-05 & lr-AdamW/pg2‚7.043014507652537e-05 trainer/global_step‚6249 % _timestamp‚1.7848506288597867e+09äÐÞÍ¬Ç & lr-AdamW/pg2‚7.043014507652537e-05 trainer/global_step‚6249 % _timestamp‚1.7848506288597867e+09 _runtime‚11392.217045001 _step‚518 & lr-AdamW/pg1‚7.043014507652537e-05†Jçâ'4­/ _wandbruntime‚11392  _runtime‚11392;<(®‚P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54265662 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1806127755436032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6521743927296 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58031477 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1806129127045632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6521743927296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16656332 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35692331 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19035999 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58041541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6523089363456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492434339705856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1799606011508736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35691975 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492433971614720 _runtime‚11392.218458524 _step‚519 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1806129127045632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7978409702400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112297139 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1799606037682176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1805406702511764 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848506288608794e+09 JNý7atsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1798895879147792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16656327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19035648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6510823363972 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58041541 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500412330050048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6512167975258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54265662 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1806127755436032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6521743927296 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58031477 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1806129127045632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6521743927296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16656332 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35692331 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19035999 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58041541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6523089363456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492434339705856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1799606011508736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35691975 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492433971614720 _runtime‚11392.218458524 _step‚519 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1806129127045632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7978409702400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112‡9ç®i4°/ _wandbruntime‚11393  _runtime‚11393~ˆ(ƒ'±ŠN D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚348 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500492162686464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 trainer/global_step‚6249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112324672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1799884946972672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _runtime‚11393.69477137 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54274035 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1799884946972672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58040559 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6522766435328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35697588 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500492582037504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7979604705792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1806409058851328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6524111878656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492512557980672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1799884920799232 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54274029 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58050637 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6522766435328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54274035 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 $ _timestamp‚1.784850630337169e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112314588 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58040559 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1799174706349536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6511844165612 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492512926071808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54274029 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58050637 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6524111878656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1806407687234560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112314588 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112324672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1799884946972672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _runtime‚11393.69477137 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54274035 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1799884946972672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58040559 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6522766435328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35697588 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500492582037504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7979604705792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1806409058851328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6524111878656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492512557980672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1799884920799232 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54274029 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58050637 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6522766435328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54274035 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 $ _timestamp‚1.784850630337169e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112314588 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58040559 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1799174706349536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6511844165612 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492512926071808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54274029 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58050637 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6524111878656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1806407687234560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112314588 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ˆÍõœì4³/ _wandbruntime‚11393  _runtime‚11393Põ ´„ 1loss/SRqqt1/detection_loss‚0.8188844688345273 epoch‚22 0loss/FRt2/assignment_loss‚0.8062374670140064 .loss/FBt2/detection_loss‚0.629976941245608 %loss/total_loss‚8.279845438971133 _runtime‚11393.73061094 0loss/FRt1/assignment_loss‚1.7002442531845197 2loss/FBt2/assignment_loss‚0.028995189605580948 1loss/SRqqt2/detection_loss‚0.6429943738771243 /loss/FBt1/detection_loss‚0.5021053744480014 trainer/global_step‚6249 % _timestamp‚1.7848506303733783e+09 3loss/FBt1/assignment_loss‚0.0013859566458272355 /loss/FRt1/detection_loss‚0.5256281892348309 /loss/FRt2/detection_loss‚1.5296205738264785 _step‚521 2loss/SRqqt1/assignment_loss‚0.5996188600330226 2loss/SRqqt2/assignment_loss‚0.4941537910216051{xhµ‰ % _timestamp‚1.7848506303733783e+09 3loss/FBt1/assignment_loss‚0.0013859566458272355 /loss/FRt1/detection_loss‚0.5256281892348309 /loss/FRt2/detection_loss‚1.5296205738264785 _step‚521 2loss/SRqqt1/assignment_loss‚0.5996188600330226 2loss/SRqqt2/assignment_loss‚0.4941537910216051 1loss/SRqqt1/detection_loss‚0.8188844688345273 epoch‚22 0loss/FRt2/assignment_loss‚0.8062374670140064 .loss/FBt2/detection_loss‚0.629976941245608 %loss/total_loss‚8.279845438971133 _runtime‚11393.73061094 0loss/FRt1/assignment_loss‚1.7002442531845197 2loss/FBt2/assignment_loss‚0.028995189605580948 1loss/SRqqt2/detection_loss‚0.6429943738771243 /loss/FBt1/detection_loss‚0.5021053744480014 trainer/global_step‚6249‰~b;ž¶‚(:“ ÌÑŠÓ·Õγ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚272.662' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.88733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/›2A·‚(:¶  ÌÑŠÓÑÇñ³ disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0$ memory_percent‚88.14597129821777 network.recv‚692506 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1550.in‚0 disk.sda.out‚ 292.83203125$ disk.nvme0n1.out‚35298.916015625 cpu‚25.325015817625605 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 88.57421875 network.sent‚ 1.1127068e+07 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 292.83203125 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0! disk.dm-0.out‚35298.916015625 proc.cpu.threads‚17 disk.drbd1584.out‚0* proc.memory.percent‚11.746978759765625) disk./.usagePercent‚70.51374563869184 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 2913.24609375 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1285.in‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.8573875427246 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 58.8583984375 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0‚åâ>¸‚(:³  ÛÑŠÓµ–ѳ network.recv‚694420 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.746978759765625 disk.sda1.out‚0% disk.nvme0n1.out‚35299.2314453125 disk.drbd1226.in‚0 disk.drbd1226.out‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1828.in‚0( proc.memory.availableMB‚ 2913.2421875 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 cpu‚25.324984211937586 network.sent‚ 1.1184862e+07 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0# memory_percent‚88.1459871927897 disk.drbd1289.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0" disk.dm-0.out‚35299.2314453125) disk./.usagePercent‚70.51374670538105$ disk./.usageGB‚1260.857406616211 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 293.0703125 disk.drbd1552.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 293.0703125 disk.drbd2111.in‚0 disk.rbd1.out‚ 58.9599609375 disk.drbd1285.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1550.in‚0Èã¸Lž¹‚(:“ ÛÑŠÓÎѬ³ gpu.0.gpu‚53.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.105' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.03500000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ù‡èº‚(:ˆ êÑŠÓª³U!ûÕÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.185' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.395 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—~~;»‚(:°  êÑŠÓ¸’ô³ disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1356.in‚0 disk.sda.out‚ 293.421875 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125* proc.memory.percent‚11.746978759765625 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 293.421875 disk.drbd1931.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 59.0224609375 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.sda1.out‚0 disk.drbd1938.in‚0( proc.memory.availableMB‚ 2913.2421875 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 proc.memory.rssMB‚ 2886.9375 cpu‚25.32486302003948 disk.drbd2078.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51374777207026 disk.drbd1552.out‚0 proc.cpu.threads‚17# memory_percent‚88.1459871927897 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚35299.4423828125 disk.drbd1673.in‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8574256896973 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0" disk.dm-0.out‚35299.4423828125 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1226.out‚0 disk.drbd1817.out‚0 network.sent‚ 1.118497e+07 network.recv‚694528 disk.drbd1965.in‚0ÎÃâÓ?¼‚(:´  ùÑŠÓ¯Öâ³ disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 293.7265625 disk.drbd1289.in‚0 disk.rbd1.out‚ 59.2021484375 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0$ disk./.usageGB‚1260.857406616211 disk.sda2.in‚0.03125 disk.sda2.out‚ 293.7265625 disk.drbd1356.out‚0* proc.memory.percent‚11.746978759765625 disk.drbd2078.out‚0( proc.memory.availableMB‚ 2913.4609375 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0$ memory_percent‚88.14509709676108 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 network.sent‚ 1.1185412e+07 disk.drbd1226.out‚0) disk./.usagePercent‚70.51374670538105" disk.dm-0.out‚35299.7861328125 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1356.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0% disk.nvme0n1.out‚35299.7861328125 proc.memory.rssMB‚ 2886.9375 disk.drbd1673.out‚0 network.recv‚695567 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1931.in‚0 cpu‚25.324787603400754 disk.drbd1550.in‚0T>ž½‚(:“ ùÑŠÓ¤Ó¿³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.553' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.51766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#35¾‚(:’ ˆÒŠÓþºÙ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.379' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.126333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V·A¿‚(:¶  ˆÒŠÓË™‹µ disk.sda2.out‚ 294.12890625 disk.drbd1356.out‚0 disk.drbd1550.in‚0 network.sent‚ 1.118552e+07% disk./.usageGB‚1260.8573684692383 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd2118.in‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.51374457200265 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0% disk.nvme0n1.out‚35299.9619140625( proc.memory.availableMB‚ 2913.4609375 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 59.2646484375 disk.drbd1285.out‚0 disk.drbd2041.out‚0$ memory_percent‚88.14509709676108 network.recv‚695675 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 88.57421875 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 cpu‚25.324749187933985 disk.drbd1938.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 294.12890625 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0" disk.dm-0.out‚35299.9619140625 disk.drbd2111.in‚0* proc.memory.percent‚11.746978759765625 disk.drbd1285.in‚0 disk.drbd1289.out‚0 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 88.57421875 proc.cpu.threads‚17 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0¼ß¹“À‚(:ˆ —ÒŠÓçïͳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.085' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.695 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mHíMBÁ‚(:·  —ÒŠÓô´ï³ disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.sda.out‚ 294.3671875 disk.dm-0.in‚ 88.57421875 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0& disk.nvme0n1.out‚35300.31884765625 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.746978759765625 network.sent‚ 1.1198939e+07) disk./.usagePercent‚70.51374777207026 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1356.in‚0 disk.drbd1828.in‚0( proc.memory.availableMB‚ 2913.2109375 cpu‚25.324657355106858 disk.drbd1226.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8574256896973 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1552.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚35300.31884765625 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.rbd1.out‚ 59.4521484375$ memory_percent‚88.14611434936523 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 network.recv‚699158 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 294.3671875‡Ž— 4Â/ _wandbruntime‚11478  _runtime‚11478”LÈàtrainer/global_step‚6299 % _timestamp‚1.7848507156007786e+09 _runtime‚11478.958095304 _step‚522 & lr-AdamW/pg1‚6.977278569266765e-05 & lr-AdamW/pg2‚6.977278569266765e-05¾FÈÍÄÇ % _timestamp‚1.7848507156007786e+09 _runtime‚11478.958095304 _step‚522 & lr-AdamW/pg1‚6.977278569266765e-05 & lr-AdamW/pg2‚6.977278569266765e-05 trainer/global_step‚6299Š½^¼»4Å/ _wandbruntime‚11478  _runtime‚11478ÿšÊ(Æ‚P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58487429 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19183357 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1819397895083372 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6561927246780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16788327 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496363658003456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16788332 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1820125781126656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54684018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1820124409517056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54684012 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35971689 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1813551476033536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6574278919680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19183002 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6572933483520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496363289912320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113181511 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504404407393280 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113171441 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6563271858066 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1813551476033536 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚504403994326528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8040704414208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚523 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1820124409517056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1819399265122626 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚6299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1813551502206976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54684018 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1812835993264560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚11478.959458676 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8040749389824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58487429 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1813551502206976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848507156019056e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1820125781126656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1812835967836592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0“È9 (LJP JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113171441 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6563271858066 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1813551476033536 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚504403994326528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8040704414208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚523 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1820124409517056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1819399265122626 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚6299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1813551502206976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54684018 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1812835993264560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚11478.959458676 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8040749389824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58487429 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1813551502206976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848507156019056e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1820125781126656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1812835967836592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58487429 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19183357 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1819397895083372 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6561927246780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16788327 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496363658003456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16788332 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1820125781126656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54684018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1820124409517056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54684012 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35971689 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1813551476033536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6574278919680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19183002 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6572933483520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496363289912320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113181511 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504404407393280 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005‹öiòx4È/ _wandbruntime‚11480  _runtime‚11480¼¢VÉŒN KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1820405712932352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16790967 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496441876278272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1813830385324032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1813114769610368 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1813830411497472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54692379 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6575301434880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1813114795038336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35977282 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8042010563584 trainer/global_step‚6299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58496511 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496442244369408 ]KDeviceStatsMonitor5Êçp.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8041955109376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚524 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1813830385324032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1819679087698154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1819677717658788 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54692379 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35976920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58506589 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113188890 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54692385 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58496511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16790972 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6564292659818 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113198974 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1813830411497472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚11480.436647709 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113198974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19185953 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6562948048420 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19186310 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573955991552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0'#(…'Ê‘N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58496511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16790972 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6564292659818 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113198974 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1813830411497472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚11480.436647709 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113198974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19185953 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6562948048420 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19186310 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573955991552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1820405712932352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16790967 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496441876278272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1813830385324032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1813114769610368 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1813830411497472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54692379 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6575301434880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1813114795038336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35977282 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8042010563584 trainer/global_step‚6299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58496511 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496442244369408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8041955109376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚524 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1813830385324032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1819679087698154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1819677717658788 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54692379 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35976920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58506589 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113188890 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54692385Œ{Àý4Ë/ _wandbruntime‚11480  _runtime‚11480‹§ Ì„ 0loss/FRt1/assignment_loss‚1.6440006209167874 2loss/FBt2/assignment_loss‚0.021254245445124727 /loss/FRt1/detection_loss‚0.5934957349090478 _runtime‚11480.473483246 1loss/SRqqt2/assignment_loss‚0.502396748646805 1loss/SRqqt1/detection_loss‚0.9457708425893326 /loss/FBt1/detection_loss‚0.6710625014478161 .loss/FBt2/detection_loss‚0.530543720200395 trainer/global_step‚6299 $ _timestamp‚1.784850717116238e+09 _step‚525 2loss/SRqqt1/assignment_loss‚0.6343400973561247 /loss/FRt2/detection_loss‚1.5003929557587137 epoch‚22 0loss/FRt2/assignment_loss‚0.7753224518914886 4loss/FBt1/assignment_loss‚0.00016632578735224554 1loss/SRqqt2/detection_loss‚0.6220227408431176 %loss/total_loss‚8.440768985792106Ž¾Yð͉ 0loss/FRt2/assignment_loss‚0.7753224518914886 4loss/FBt1/assignment_loss‚0.00016632578735224554 1loss/SRqqt2/detection_loss‚0.6220227408431176 %loss/total_loss‚8.440768985792106 0loss/FRt1/assignment_loss‚1.6440006209167874 2loss/FBt2/assignment_loss‚0.021254245445124727 /loss/FRt1/detection_loss‚0.5934957349090478 _runtime‚11480.473483246 1loss/SRqqt2/assignment_loss‚0.502396748646805 1loss/SRqqt1/detection_loss‚0.9457708425893326 /loss/FBt1/detection_loss‚0.6710625014478161 .loss/FBt2/detection_loss‚0.530543720200395 trainer/global_step‚6299 $ _timestamp‚1.784850717116238e+09 _step‚525 2loss/SRqqt1/assignment_loss‚0.6343400973561247 /loss/FRt2/detection_loss‚1.5003929557587137 epoch‚22÷!qŞ΂(:“ ¦ÒŠÓ“µÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.87' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.95666666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µŠ}ûDÏ‚(:¹  ¦ÒŠÓٻೠdisk.drbd1289.in‚0 disk.sda1.in‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0$ memory_percent‚88.14624150594076) disk./.usagePercent‚70.51374777207026 disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 network.recv‚700196 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0& disk.nvme0n1.out‚35300.47900390625 cpu‚25.324560722229858 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1550.in‚0* proc.memory.percent‚11.746978759765625 disk.drbd1356.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1510.in‚0 disk.sda.out‚ 294.75390625 proc.memory.rssMB‚ 2886.9375 disk.drbd1817.in‚0( proc.memory.availableMB‚ 2913.1796875 disk.rbd1.out‚ 59.5146484375 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1965.in‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 network.sent‚ 1.1199908e+07% disk./.usageGB‚1260.8574256896973 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 294.75390625# disk.dm-0.out‚35300.47900390625 disk.drbd2078.out‚0 disk.drbd1285.out‚0ª†¼ žÐ‚(:“ µÒŠÓ†É³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.102' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.70066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 `(È>Ñ‚(:³  µÒŠÓó·æ³# memory_percent‚92.1014149983724 disk.sda2.out‚ 295.21484375" disk.dm-0.out‚35301.5107421875% disk.nvme0n1.out‚35301.5107421875 proc.memory.rssMB‚ 2886.9375 cpu‚25.324506909478504 disk.drbd1289.out‚0 network.sent‚ 1.125995e+07 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 59.5146484375* proc.memory.percent‚11.746978759765625) disk./.usagePercent‚70.51374734539458 disk.sda1.in‚0 disk.sda.out‚ 295.21484375 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0& proc.memory.availableMB‚ 1941.15625 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0% disk./.usageGB‚1260.8574180603027 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 88.57421875 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 network.recv‚702817 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0Ô¿¾;Ò‚(:°  ÄÒŠÓ‚Äè³ disk.drbd1550.out‚0 disk.drbd1584.out‚0 cpu‚25.32443173752586 proc.cpu.threads‚17 network.sent‚ 1.1262944e+07 disk.dm-0.in‚ 88.57421875 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125% disk.nvme0n1.out‚35301.8857421875) disk./.usagePercent‚70.51374627870537 disk.drbd2111.out‚0 disk.sda.out‚ 295.44921875 proc.memory.rssMB‚ 2886.9375 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0% disk./.usageGB‚1260.8573989868164 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0$ memory_percent‚92.10243225097656 disk.rbd1.out‚ 59.546875 network.recv‚703885 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 295.44921875& proc.memory.availableMB‚ 1940.90625* proc.memory.percent‚11.746978759765625 disk.drbd1226.in‚0" disk.dm-0.out‚35301.8857421875 disk.drbd1828.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125Uõ¾ÐžÓ‚(:“ ÄÒŠÓÎîij gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚271.107' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.36900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s0¨ûŸÔ‚(:” ÓÒŠÓ²“°³ gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.895' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.29833333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 *=8Õ‚(:­  ÓÒŠÓ‰æγ disk.sda.out‚ 295.796875 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0( disk./.usagePercent‚70.5135500078918$ disk./.usageGB‚1260.853889465332 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 295.796875 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0* proc.memory.percent‚11.746978759765625 network.sent‚ 1.1265431e+07 disk.drbd1550.out‚0 proc.memory.rssMB‚ 2886.9375 disk.dm-0.in‚ 88.57421875# disk.dm-0.out‚35302.37646484375 disk.drbd1931.in‚0 disk.drbd2041.in‚0& proc.memory.availableMB‚ 1940.90625 proc.cpu.threads‚17 disk.drbd1673.in‚0 network.recv‚704972 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 cpu‚25.32444420176268 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.10243225097656 disk.sda1.in‚0 disk.rbd1.out‚ 59.6484375 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚35302.37646484375iÀIƒÖ‚(:’ âÒŠÓàÞÁ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.44' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.81333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÿKU4ׂ(:©  âÒŠÓ»Žá³ disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 cpu‚25.32432556583871 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.rbd1.in‚0* proc.memory.percent‚11.746978759765625$ disk./.usageGB‚1260.853889465332 disk.drbd1817.out‚0 disk.sda2.out‚296.0625 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.dm-0.in‚ 88.57421875& disk.nvme0n1.out‚35302.50146484375 disk.drbd1510.out‚0 proc.memory.rssMB‚ 2886.9375( disk./.usagePercent‚70.5135500078918 disk.sda.out‚296.0625 disk.drbd1828.out‚0$ memory_percent‚92.10243225097656 disk.drbd1584.in‚0 network.sent‚ 1.1270785e+07 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 59.7109375& proc.memory.availableMB‚ 1940.90625 network.recv‚705302 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1510.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚35302.50146484375 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0Þ hïžØ‚(:“ ñÒŠÓÊÆä³ gpu.0.gpu‚89.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚254.667' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.88900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âžô\;Ù‚(:°  ñÒŠÓÛ„´ network.recv‚707100 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1938.out‚0$ memory_percent‚92.10243225097656 proc.memory.rssMB‚ 2886.9375 disk.drbd1931.in‚0& disk.nvme0n1.out‚35302.93212890625* proc.memory.percent‚11.746978759765625 disk.sda2.out‚ 296.671875 disk.drbd1289.in‚0# disk.dm-0.out‚35302.93212890625 disk.drbd2111.in‚0 disk.rbd1.out‚ 59.7109375 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 network.sent‚ 1.1273999e+07% disk./.usageGB‚1260.8538513183594 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51354787451338 disk.sda1.in‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1285.in‚0& proc.memory.availableMB‚ 1940.90625 disk.drbd1510.in‚0 cpu‚25.324271306637343 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 296.671875 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17-+Ã4Ú/ _wandbruntime‚11565  _runtime‚11565^¦HÈÛ & lr-AdamW/pg1‚6.911395716086012e-05 & lr-AdamW/pg2‚6.911395716086012e-05 trainer/global_step‚6349 % _timestamp‚1.7848508025345964e+09 _runtime‚11565.892888164 _step‚526ÿ4YÍÜÇ _step‚526 & lr-AdamW/pg1‚6.911395716086012e-05 & lr-AdamW/pg2‚6.911395716086012e-05 trainer/global_step‚6349 % _timestamp‚1.7848508025345964e+09 _runtime‚11565.892888164Žûè 4Ý/ _wandbruntime‚11565  _runtime‚11565梽AÛÞ‚P [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6625468475904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1834121063598080 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58953445 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508396242785792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1834121063598080 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allo ‚ò·- cated‚1827496966731776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55102368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58953445 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6613031129588 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1834122435207680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58943381 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36251706 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6624123039744 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1827496940558336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55102362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500292608209920 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1833390457694234 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16920332 _runtime‚11565.892993432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1827496940558336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55102362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500292608209920 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1833390457694234 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16920332 _runtime‚11565.892993432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58953445 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508396242785792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1834121063598080 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1827496966731776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55102368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58953445 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6613031129588 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1834122435207680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58943381 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36251706 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6624123039744 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36257338 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58962541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508475660555264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1827775849848832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114063192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55110729 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58952463 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8104515142144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114073276 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55110735 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1827775849848832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55110729 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508476077809152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6625145547776 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1834402367013376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1827054858299168 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1834400995396608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55110735 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36257694 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19334371 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16922967 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114063192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6614051931228 % _timestamp‚1.7848508040128956e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58952463 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1827775876022272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6625145547776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16922972 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8104465979392]„)]â‘N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19334371 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16922967 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114063192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6614051931228 % _timestamp‚1.7848508040128956e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58952463 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1827775876022272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6625145547776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16922972 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8104465979392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1833670280269762 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36257338 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58962541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508475660555264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1827775849848832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114063192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55110729 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58952463 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8104515142144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114073276 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55110735 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1827775849848832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55110729 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508476077809152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6625145547776 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1834402367013376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1827054858299168 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1834400995396608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55110735 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36257694 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248?9i 4ã/ _wandbruntime‚11567  _runtime‚11567$gwb ä… /loss/FRt1/detection_loss‚0.5529700924682063 /loss/FRt2/detection_loss‚1.5322354013565929 1loss/SRqqt1/detection_loss‚0.8561103709679047 /loss/FBt2/detection_loss‚0.3426772855077493 3loss/FBt1/assignment_loss‚0.0029470874623337786 epoch‚22 trainer/global_step‚6349 % _timestamp‚1.7848508040488355e+09 _runtime‚11567.406074183 _step‚529 0loss/FRt1/assignment_loss‚1.7328537868533849 0loss/FRt2/assignment_loss‚0.7740182746024359 2loss/FBt2/assignment_loss‚0.031713857557179405 1loss/SRqqt2/detection_loss‚0.6462410664752971 .loss/FBt1/detection_loss‚0.556816131186982 2loss/SRqqt1/assignment_loss‚0.5283695077850885 2loss/SRqqt2/assignment_loss‚0.5614527851617975 %loss/total_loss‚8.118405647384952²McåŠ % _timestamp‚1.7848508040488355e+09 _runtime‚11567.406074183 _step‚529 0loss/FRt1/assignment_loss‚1.7328537868533849 0loss/FRt2/assignment_loss‚0.7740182746024359 2loss/FBt2/assignment_loss‚0.031713857557179405 1loss/SRqqt2/detection_loss‚0.6462410664752971 .loss/FBt1/detection_loss‚0.556816131186982 2loss/SRqqt1/assignment_loss‚0.5283695077850885 2loss/SRqqt2/assignment_loss‚0.5614527851617975 %loss/total_loss‚8.118405647384952 /loss/FRt1/detection_loss‚0.5529700924682063 /loss/FRt2/detection_loss‚1.5322354013565929 1loss/SRqqt1/detection_loss‚0.8561103709679047 /loss/FBt2/detection_loss‚0.3426772855077493 3loss/FBt1/assignment_loss‚0.0029470874623337786 epoch‚22 trainer/global_step‚6349‘¼xŽ^Ÿæ‚(:” €ÓŠÓߊ¸³ gpu.0.gpu‚57.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚304.935' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.64500000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äæ|¾@ç‚(:µ  €ÓŠÓíôÕ³ disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 network.recv‚708096 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 59.7431640625& proc.memory.availableMB‚ 1940.84375 disk.sda.in‚0.03125 disk.drbd2111.out‚0* proc.memory.percent‚11.746978759765625$ disk./.usageGB‚1260.854290008545# disk.dm-0.out‚35303.15869140625 disk.nvme0n1.in‚ 88.57421875 network.sent‚ 1.1275075e+07 disk.drbd1584.in‚0 disk.sda2.out‚ 296.91796875 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1356.out‚0 proc.memory.rssMB‚ 2886.9375 disk.dm-0.in‚ 88.57421875& disk.nvme0n1.out‚35303.15869140625) disk./.usagePercent‚70.51357240836508 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 cpu‚25.324198557885204 disk.sda.out‚ 296.91796875 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0# memory_percent‚92.1026865641276 disk.drbd1938.out‚0XÂ)•@è‚(:µ  ÓŠÓÕ®ì³% disk./.usageGB‚1260.8543090820312 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 297.2265625 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 59.8447265625 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1356.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 cpu‚25.32414502965695 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125" disk.dm-0.out‚35303.4912109375 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1940.83984375 proc.memory.rssMB‚ 2886.9375 proc.cpu.threads‚17 network.sent‚ 1.1337365e+07) disk./.usagePercent‚70.51357347505429 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 88.57421875% disk.nvme0n1.out‚35303.4912109375 disk.drbd2118.in‚0 disk.drbd1289.in‚0$ memory_percent‚92.10270245869955 network.recv‚711100 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 297.2265625 disk.drbd1285.out‚0* proc.memory.percent‚11.746978759765625ùŒÊ¿žé‚(:“ ÓŠÓˆñɳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.415' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.13833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šï?ê‚(:´  žÓŠÓ–’ݳ disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.746978759765625 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0) disk./.usagePercent‚70.51357347505429 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1938.in‚0& proc.memory.availableMB‚ 1940.84375 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 297.91796875" disk.dm-0.out‚35303.7177734375% disk.nvme0n1.out‚35303.7177734375 disk.sda1.out‚0 disk.drbd1584.in‚0 cpu‚25.324071527467826 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 proc.memory.rssMB‚ 2886.9375# memory_percent‚92.1026865641276 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 59.9072265625 disk.nvme0n1.in‚ 88.57421875 disk.sda1.in‚0 network.sent‚ 1.1339641e+07% disk./.usageGB‚1260.8543090820312 disk.drbd2111.out‚0 disk.drbd1285.out‚0 network.recv‚711298 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 297.91796875 disk.dm-0.in‚ 88.57421875 disk.drbd1965.in‚0 disk.drbd2041.out‚0¹"¦žë‚(:“ žÓŠÓ¡¼¿³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.403' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.13433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ifÊžì‚(:“ ­ÓŠÓåˆ×³ gpu.0.gpu‚45.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.488' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.16266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TTÃ!-í‚(:¢  ­ÓŠÓöö³ disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.sda2.in‚0.03125 proc.cpu.threads‚17 network.sent‚ 1.1339749e+07 disk.drbd1226.in‚0 disk.sda.out‚ 298.5078125 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0" disk.nvme0n1.out‚ 35304.0390625 disk.drbd1817.in‚0 disk.drbd1226.out‚0$ disk./.usageGB‚1260.854290008545 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 cpu‚25.32401822063188 disk.drbd1510.in‚0 disk.sda2.out‚ 298.5078125 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd2041.in‚0 disk.sda.in‚0.03125! memory_percent‚92.10205078125 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.746978759765625 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 network.recv‚711406 disk.drbd1285.in‚0 disk.dm-0.out‚ 35304.0390625 disk.rbd1.out‚ 60.1806640625 disk.drbd1552.out‚0 proc.memory.availableMB‚1941 proc.memory.rssMB‚ 2886.9375) disk./.usagePercent‚70.51357240836508 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0n÷&B8î‚(:­  ¼ÓŠÓ¤êÕ³ disk.drbd2078.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.sda.in‚0.03125 disk.rbd1.out‚ 60.3291015625 disk.drbd1550.out‚0 network.sent‚ 1.1339815e+07 disk.rbd1.in‚0! memory_percent‚92.10205078125 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd2111.in‚0 disk.drbd1226.in‚0# disk.dm-0.out‚35304.29052734375 disk.drbd2118.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.out‚ 299.08984375 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0$ disk./.usageGB‚1260.854232788086 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 proc.memory.rssMB‚ 2886.9375) disk./.usagePercent‚70.51356920829747 disk.drbd2041.in‚0* proc.memory.percent‚11.746978759765625 cpu‚25.323924275757633 disk.sda2.out‚ 299.08984375 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0 proc.memory.availableMB‚1941 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0& disk.nvme0n1.out‚35304.29052734375 network.recv‚711472 disk.nvme0n1.in‚ 88.57421875 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚055÷žï‚(:“ ¼ÓŠÓ¢»³³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.516' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.83866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0} 4ð/ _wandbruntime‚11652  _runtime‚11652S,t7Èñ % _timestamp‚1.7848508893918161e+09 _runtime‚11652.749070442 _step‚530 & lr-AdamW/pg1‚6.845374583842881e-05 & lr-AdamW/pg2‚6.845374583842881e-05 trainer/global_step‚6399Ü"ÉÍòÇ _runtime‚11652.749070442 _step‚530 & lr-AdamW/pg1‚6.845374583842881e-05 & lr-AdamW/pg2‚6.845374583842881e-05 trainer/global_step‚6399 % _timestamp‚1.7848508893918161e+09’‰2"‡4ó/ _wandbruntime‚11652  _runtime‚11652–˜ž¾(ô‚P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1841442431256576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114930115 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59409397 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1841442431256576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59399333 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6676658032128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36532058 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1847381650265842 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1848117717679104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1848117717679104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1847380280226588 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55520712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19480096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1840716145214192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36532428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6664135012396 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚531 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1841442405083136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512387641662464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504222294598656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504221926507520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8165768519168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1840716170642160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 % _timestamp‚1.7848508893928943e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59399333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8165715154944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114920045 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512388063117824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675312595968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1848117717679104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1848117717679104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1847380280226588 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55520712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19480096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1840716145214192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36532428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6664135012396 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚531 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1841442405083136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512387641662464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504222294598656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504221926507520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8165768519168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1840716170642160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 % _timestamp‚1.7848508893928943e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59399333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8165715154944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114920045 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512388063117824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675312595968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1841721314373632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114947578 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55529079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1847660102802004 _runtime‚11654.224700486 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8167022694912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17054972 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55529085 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512467484308480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1848399021094400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512467903659520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504300512873472 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1840994946987968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1841721340547072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59408415 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848508908671117e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1848397649477632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 trainer/global_step‚6399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1841721314373632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36538023 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114937494 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114937494 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17054967 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1840994972415936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114947578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59418493 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚363 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6677680547328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6676335104000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6677680547328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6676335104000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55529085 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36537660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1841721340547072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0%WƒD'ø‘N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512467903659520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504300512873472 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1840994946987968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1841721340547072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59408415 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848508908671117e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1848397649477632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 trainer/global_step‚6399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1841721314373632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36538023 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114937494 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114937494 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17054967 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1840994972415936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114947578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59418493 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚363 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6677680547328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6676335104000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6677680547328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6676335104000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55529085 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36537660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1841721340547072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55529079 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1847661472841370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19483051 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166971435008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6666500425434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59408415 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1848399021094400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚358 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1841721314373632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114947578 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55529079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1847660102802004 _runtime‚11654.224700486 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8167022694912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17054972 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55529085 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512467484308480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1848399021094400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0”Àð204ù/ _wandbruntime‚11654  _runtime‚11654)…X•ú 2loss/SRqqt1/assignment_loss‚0.5574122362352084 /loss/FRt1/detection_loss‚0.5970787968816276 _runtime‚11654.263196842 _step‚533 2loss/SRqqt2/assignment_loss‚0.5585948349639216 1loss/FBt2/assignment_loss‚0.02818078164470819 .loss/FBt2/detection_loss‚0.556596058118157 %loss/total_loss‚8.558196995389455 0loss/FRt2/assignment_loss‚0.8135885541190468 2loss/FBt1/assignment_loss‚0.002978203745215261 .loss/FRt2/detection_loss‚1.431028133364156 1loss/SRqqt1/detection_loss‚0.9770964898535165 1loss/SRqqt2/detection_loss‚0.6524776043682308 epoch‚22 $ _timestamp‚1.784850890905977e+09 0loss/FRt1/assignment_loss‚1.7151581912712106 /loss/FBt1/detection_loss‚0.6680071108244561 trainer/global_step‚6399uv˜( û† 0loss/FRt1/assignment_loss‚1.7151581912712106 /loss/FBt1/detection_loss‚0.6680071108244561 trainer/global_step‚6399 2loss/SRqqt1/assignment_loss‚0.5574122362352084 /loss/FRt1/detection_loss‚0.5970787968816276 _runtime‚11654.263196842 _step‚533 2loss/SRqqt2/assignment_loss‚0.5585948349639216 1loss/FBt2/assignment_loss‚0.02818078164470819 .loss/FBt2/detection_loss‚0.556596058118157 %loss/total_loss‚8.558196995389455 0loss/FRt2/assignment_loss‚0.8135885541190468 2loss/FBt1/assignment_loss‚0.002978203745215261 .loss/FRt2/detection_loss‚1.431028133364156 1loss/SRqqt1/detection_loss‚0.9770964898535165 1loss/SRqqt2/detection_loss‚0.6524776043682308 epoch‚22 $ _timestamp‚1.784850890905977e+09•–wŸü‚(:” ËÓŠÓÆõг gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚131.702' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚43.900666666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a®ó(Eý‚(:º  ËÓŠÓù¸ó³ disk.drbd1550.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd2078.in‚0 disk.sda.out‚ 299.50390625 disk.sda2.in‚0.03125) proc.memory.availableMB‚ 1941.24609375% disk./.usageGB‚1260.8543128967285 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0# disk.dm-0.out‚35304.89794921875 disk.drbd2111.out‚0 disk.drbd1285.in‚0$ memory_percent‚92.10104942321777 disk.sda1.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚35304.89794921875 disk.drbd2111.in‚0 disk.rbd1.out‚ 60.4306640625 disk.drbd1285.out‚0 proc.memory.rssMB‚ 2886.9375 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 299.50390625 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 88.57421875* proc.memory.percent‚11.746978759765625 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 cpu‚25.323849739317907) disk./.usagePercent‚70.51357368839213 disk.drbd1817.out‚0 disk.drbd1965.out‚0 network.sent‚ 1.1342784e+07 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 network.recv‚713323 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125wqËEAþ‚(:¶  ÚÓŠÓËá„´ disk.sda.in‚0.03125 disk.dm-0.in‚ 88.57421875 disk.drbd1828.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 299.80078125 disk.drbd1550.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 1940.96875 disk.drbd2118.out‚0% disk./.usageGB‚1260.8543701171875 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.nvme0n1.in‚ 88.57421875& disk.nvme0n1.out‚35305.50146484375 disk.sda2.in‚0.03125 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0# disk.dm-0.out‚35305.50146484375 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd2111.out‚0$ memory_percent‚92.10217793782553 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1342784e+07 disk.drbd1356.in‚0 disk.rbd1.out‚ 60.4931640625 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.746978759765625 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 cpu‚25.32381855931029) disk./.usagePercent‚70.51357688845974 disk.sda2.out‚ 299.80078125 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 proc.memory.rssMB‚ 2886.9375 network.recv‚713323 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0ÎÊçaŸÿ‚(:” ÚÓŠÓ»èä³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚313.838' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.61266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DˆŸ€‚(:” éÓŠÓÔà±³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.904' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.30133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï3ËS¼‚(:¬  éÓŠÓÙðÕ³ disk.drbd1356.in‚0 disk.rbd1.out‚ 60.4931640625 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 proc.memory.rssMB‚ 2886.9375 proc.cpu.threads‚17 network.sent‚ 1.1414094e+07 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.dm-0.out‚ 35306.6015625 disk.rbd1.in‚0 disk.sda.out‚ 300.359375 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.dm-0.in‚ 88.57421875(/þâ/{ disk./.usagePercent‚70.5135777418111% disk./.usageGB‚1260.8543853759766 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.746978759765625 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 300.359375 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 cpu‚25.32374705663184 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd2041.in‚0" disk.nvme0n1.out‚ 35306.6015625) proc.memory.availableMB‚ 1940.96484375 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0$ memory_percent‚92.10219383239746 disk.drbd1550.out‚0 network.recv‚717398‚øqâ8‚‚(:­  øÓŠÓñƒá³ disk.dm-0.out‚35307.08203125 disk.sda.in‚0.03125 disk.drbd2118.out‚0* proc.memory.percent‚11.746978759765625 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚92.1013037363688 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 network.recv‚717464 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 300.765625 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1552.in‚0 network.sent‚ 1.141416e+07 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 300.765625% disk./.usageGB‚1260.8543090820312 disk.drbd1938.in‚0# disk.nvme0n1.out‚35307.08203125 disk.drbd1931.out‚0 cpu‚25.323694998784223 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 88.57421875 disk.rbd1.out‚ 60.525390625 disk.drbd1226.in‚0) disk./.usagePercent‚70.51357347505429 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0) proc.memory.availableMB‚ 1941.18359375 proc.memory.rssMB‚ 2886.9375 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0ØÚ5žƒ‚(:“ øÓŠÓ”ÙÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚249.946' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.31533333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ F“„‚(:ˆ ‡ÔŠÓµžÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.469' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.823 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒŒL]>…‚(:³  ‡ÔŠÓË—õ³ disk.drbd2078.in‚0% disk.nvme0n1.out‚35307.6298828125 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 60.626953125 disk.drbd1550.out‚0* proc.memory.percent‚11.746978759765625 disk.drbd1552.out‚0 disk.drbd1226.in‚0 network.recv‚718482 disk.sda.out‚ 300.9921875 disk.dm-0.in‚ 88.57421875 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1940.93359375% disk./.usageGB‚1260.8562812805176 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.rbd1.in‚0$ memory_percent‚92.10232098897299 disk.sda1.in‚0 disk.drbd1584.out‚0" disk.dm-0.out‚35307.6298828125 disk.drbd1965.in‚0 cpu‚25.323621475245133 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1510.out‚0' disk./.usagePercent‚70.513683770718 disk.drbd1289.out‚0 disk.drbd1817.out‚0 proc.memory.rssMB‚ 2886.9375 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.sda2.out‚ 300.9921875 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 network.sent‚ 1.1415202e+07 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0%ZÓxŸ†‚(:” –ÔŠÓˆù·³ gpu.0.gpu‚96.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.352' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.78399999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±>qE‡‚(:º  –ÔŠÓ“¸Ù³ disk.drbd1584.in‚0 disk.drbd1965.in‚0 network.recv‚721370 disk.drbd2111.in‚0 disk.rbd1.out‚ 60.689453125 disk.drbd1552.in‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 301.39453125 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 cpu‚25.324739940225875 network.sent‚ 1.1423371e+07) disk./.usagePercent‚70.51368313070448 disk.drbd1552.out‚0$ memory_percent‚92.11859703063965 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8562698364258 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0" disk.dm-0.out‚35307.8291015625 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.sda2.in‚0.03125 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚35307.8291015625 disk.drbd1673.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd2118.out‚0) proc.memory.availableMB‚ 1936.93359375 disk.drbd1828.in‚0 disk.sda.out‚ 301.39453125 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 88.57421875" proc.memory.rssMB‚ 2890.8828125 proc.cpu.threads‚17* proc.memory.percent‚11.763032277425129 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0Òj-œ4ˆ/ _wandbruntime‚11742  _runtime‚11742K­…M$;‰žv particle/accuracy_2_of_3‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.761408269405365 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.812751293182373 'particle/f_score‚0.6521855592727661 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.584064781665802 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6208475828170776 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0281513798981905 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3232763409614563 (particle/accuracy‚0.8538120985031128 _runtime‚11742.845815786 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45653003454208374 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018937574699521065 % _timestamp‚1.7848509794877205e+09 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40017127990722656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963475465774536 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 particle/accuracy_1_of_2‚1 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763218879699707 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7876300811767578 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14623352885246277 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6940674185752869 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7511186003684998 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992559552192688 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7068652510643005 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚22 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 particle/accuracy_1_of_4‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11323587596416473 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3417939841747284 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6844450831413269 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6182302832603455 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0370015986263752 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974352121353149 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24428169429302216 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8116270303726196 )jet/accuracy_2_of_3‚0.597140371799469 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 *jet/accuracy_2_of_4‚0.8258516192436218 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6669579744338989 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13821910321712494 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24038876593112946 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.723664402961731 *jet/accuracy_1_of_3‚0.9369398355484009 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888365864753723 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.713821530342102 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9125329852104187 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7538042068481445 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4916069209575653 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27485260367393494 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7585927844047546 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16707749664783478 trainer/global_step‚6439 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8035510182380676 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6888248324394226 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13024988770484924 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3232763409614563 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920055270195007 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 /particle/accuracy_3_of_4‚0.9913265109062195 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5409945845603943 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27057555317878723 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8002988696098328 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3222992420196533 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7931046485900879 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40022942423820496 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27634936571121216 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7497917413711548 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8713331818580627 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6639589667320251 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1168898418545723 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944790005683899 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6731661558151245 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7350624203681946 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2619010806083679 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6731661558151245 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07332458347082138 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 *jet/accuracy_1_of_2‚0.7956725358963013 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.99146568775177 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.482020765542984 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5761553645133972 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8537949323654175 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6704467535018921 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2645716071128845 *jet/accuracy_3_of_3‚0.0696956142783165 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3360414206981659 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9230343103408813 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 )loss/val_total_loss‚8.301588895941574 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8037647008895874 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2586529552936554 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46077296137809753 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6445560455322266 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.644597053527832 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6962170600891113 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6265981793403625 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_2_of_4‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9972655177116394 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9983359575271606 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41886574029922485 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 +particle/sensitivity‚0.6388376355171204 _step‚534 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7024819254875183 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5351455211639404 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05368087440729141 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7582646608352661 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8713331818580627 +jet/accuracy_1_of_1‚0.36905166506767273 +jet/accuracy_3_of_4‚0.11328570544719696 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9977060556411743 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.33502456545829773 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7837817072868347 *jet/accuracy_1_of_4‚0.9758731126785278 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7186122536659241 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018937574699521065 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029719997197389603 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6764254570007324 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3354024589061737 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6552726626396179 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 /particle/accuracy_4_of_4‚0.9600567817687988 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3435210883617401 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7999382019042969 6validation_average_jet_accuracy‚0.4035816788673401 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15167073905467987 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7587398886680603 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7255996465682983 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.263168066740036 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37548714876174927 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8640095591545105 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21289873123168945 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25653520226478577 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3760457932949066 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24656757712364197 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03485717251896858 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05619976297020912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7519311308860779 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9223038554191589 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7430020570755005 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20952937006950378 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04590052738785744 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.30881282687187195 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.31454235315322876 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7514842748641968 ¯I\)Š£v & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4916069209575653 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27485260367393494 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7585927844047546 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16707749664783478 trainer/global_step‚6439 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8035510182380676 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6888248324394226 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13024988770484924 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3232763409614563 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920055270195007 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 /particle/accuracy_3_of_4‚0.9913265109062195 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5409945845603943 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27057555317878723 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8002988696098328 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3222992420196533 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7931046485900879 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40022942423820496 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27634936571121216 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7497917413711548 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8713331818580627 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6639589667320251 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1168898418545723 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944790005683899 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6731661558151245 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7350624203681946 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2619010806083679 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6731661558151245 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07332458347082138 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 *jet/accuracy_1_of_2‚0.7956725358963013 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.99146568775177 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.482020765542984 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5761553645133972 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8537949323654175 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6704467535018921 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2645716071128845 *jet/accuracy_3_of_3‚0.0696956142783165 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3360414206981659 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9230343103408813 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 )loss/val_total_loss‚8.301588895941574 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8037647008895874 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2586529552936554 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46077296137809753 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6445560455322266 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.644597053527832 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6962170600891113 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6265981793403625 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_2_of_4‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9972655177116394 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9983359575271606 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41886574029922485 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 +particle/sensitivity‚0.6388376355171204 _step‚534 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7024819254875183 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5351455211639404 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05368087440729141 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7582646608352661 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8713331818580627 +jet/accuracy_1_of_1‚0.36905166506767273 +jet/accuracy_3_of_4‚0.11328570544719696 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991496801376343 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9977060556411743 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.33502456545829773 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7837817072868347 *jet/accuracy_1_of_4‚0.9758731126785278 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7186122536659241 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018937574699521065 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029719997197389603 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6764254570007324 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3354024589061737 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6552726626396179 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 /particle/accuracy_4_of_4‚0.9600567817687988 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3435210883617401 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7999382019042969 6validation_average_jet_accuracy‚0.4035816788673401 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15167073905467987 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7587398886680603 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7255996465682983 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.263168066740036 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998964786529541 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37548714876174927 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8640095591545105 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21289873123168945 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25653520226478577 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3760457932949066 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24656757712364197 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03485717251896858 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05619976297020912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7519311308860779 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9223038554191589 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7430020570755005 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20952937006950378 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04590052738785744 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.30881282687187195 :"PuÙ´ã-Írity/*FRt0SRqqt0FBt/event_purity‚0.31454235315322876 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7514842748641968 particle/accuracy_2_of_3‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.761408269405365 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.812751293182373 'particle/f_score‚0.6521855592727661 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.584064781665802 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6208475828170776 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0281513798981905 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3232763409614563 (particle/accuracy‚0.8538120985031128 _runtime‚11742.845815786 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45653003454208374 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018937574699521065 % _timestamp‚1.7848509794877205e+09 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40017127990722656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963475465774536 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998964786529541 particle/accuracy_1_of_2‚1 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763218879699707 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7876300811767578 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14623352885246277 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6940674185752869 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7511186003684998 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992559552192688 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7068652510643005 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚22 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 particle/accuracy_1_of_4‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11323587596416473 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3417939841747284 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6844450831413269 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6182302832603455 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0370015986263752 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974352121353149 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24428169429302216 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8116270303726196 )jet/accuracy_2_of_3‚0.597140371799469 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 *jet/accuracy_2_of_4‚0.8258516192436218 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6669579744338989 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13821910321712494 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24038876593112946 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.723664402961731 *jet/accuracy_1_of_3‚0.9369398355484009 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888365864753723 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.713821530342102 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9125329852104187 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7538042068481445 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1–>Õ}K‘‹‚(:† ¥ÔŠÓˆ„¿³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚58.0 gpu.0.powerWatts‚105.867' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.289 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆP¼ã=Œ‚(:²  ¥ÔŠÓæ•Þ³ network.recv‚722490 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 cpu‚25.32748873706092 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.03125" disk.dm-0.out‚35308.3193359375 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚35308.3193359375 proc.cpu.threads‚17% disk./.usageGB‚1262.8108367919922 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1552.out‚0! proc.memory.rssMB‚ 3863.390625) disk./.usagePercent‚70.62299274692302 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0% proc.memory.availableMB‚ 1.5546875 disk.drbd1673.out‚0 disk.sda2.out‚ 301.9921875 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 301.9921875 disk.drbd1817.in‚0 network.sent‚ 1.1426361e+07 disk.drbd2111.in‚0 disk.rbd1.out‚ 60.689453125 disk.drbd1817.out‚0$ memory_percent‚99.99367396036783 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd2118.out‚0* proc.memory.percent‚15.720176696777344o¹›t“‚(:ˆ ´ÔŠÓÁ§¹³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚276.111' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.037 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Ã}DŽ‚(:¹  ´ÔŠÓÎåݳ disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 88.57421875 network.recv‚724314 disk.drbd1938.in‚0 disk.rbd1.out‚ 60.9013671875 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 302.27734375 disk.drbd1510.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚37247.5771484375 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 network.sent‚ 1.1480144e+07) disk./.usagePercent‚70.51368206401528 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0$ memory_percent‚88.13079198201498 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2893.89453125) proc.memory.percent‚11.77528699239095 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚37247.5771484375 disk.drbd1550.out‚0( proc.memory.availableMB‚ 2916.9765625 disk.sda1.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 cpu‚25.32626670738866 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.drbd2041.in‚0% disk./.usageGB‚1260.8562507629395 disk.drbd2111.out‚0 disk.sda.out‚ 302.27734375 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1828.in‚0⢽s4/ _wandbruntime‚11763  _runtime‚11763”µŠKÈ trainer/global_step‚6449 % _timestamp‚1.7848510006045797e+09 _runtime‚11763.961822898 _step‚535 & lr-AdamW/pg1‚6.779223826395185e-05 & lr-AdamW/pg2‚6.779223826395185e-05ù}aùÍ‘Ç _runtime‚11763.961822898 _step‚535 & lr-AdamW/pg1‚6.779223826395185e-05 & lr-AdamW/pg2‚6.779223826395185e-05 trainer/global_step‚6449 % _timestamp‚1.7848510006045797e+09—™,Õ4’/ _wandbruntime‚11763  _runtime‚11763¼ƒÂ5(“P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17191219 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚516435567855616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508200612409344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1864680666924032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56028500 _runtime‚11763.963092429 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1857948546332672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1857215235054160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59912661 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8234920956416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6722192345804 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1864680666924032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402580992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56028500 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19655893 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115931091 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1857948546332672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59902597 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56028494 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1864682038533632 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56028494 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36847373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8234955446272 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6733466027520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1864680666924032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56028500 _runtime‚11763.963092429 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1857948546332672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1857215235054160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59912661 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8234920956416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6722192345804 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1864680666924032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402580992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56028500 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19655893 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115931091 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1857948546332672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59902597 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56028494 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1864682038533632 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56028494 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36847373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8234955446272 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6733466027520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36852767 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1864217275403460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115958624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56036861 RR EÊ#CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1858227481796608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115948540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17193859 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚516515067403776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6723213147556 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6733143099392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚508279198775296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59921757 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1858227481796608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6734488542720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11765.435112153 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56036867 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6721868536158 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115958624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8236236719616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6734488542720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6733143099392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19658908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508278830684160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8236271202304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115948540 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚508279198775296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59921757 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1858227481796608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6734488542720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11765.435112153 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56036867 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6721868536158 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115958624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8236236719616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6734488542720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6733143099392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19658908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508278830684160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8236271202304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115948540 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36852767 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1864217275403460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115958624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56036861 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1858227481796608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115948540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17193859 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚516515067403776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6723213147556™ÐÓDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1871894037030912 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1871894010857472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17323224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6784655583744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1878678692614656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1871894010857472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚520427800091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6783310147584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚512129930706944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56446850 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1878677321005056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1878677321005056 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520428221546496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8298290839552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824ùåÀs (¬‡P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6771951617326 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17323219 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56446844 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17323224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6784655583744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1878678692614656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1871894010857472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚520427800091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6783310147584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚512129930706944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56446850 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1878677321005056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1878677321005056 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520428221546496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8298290839552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60358549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19805164 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1877927275360286 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1871894037030912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116815463 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6783310147584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19805506 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1877928645399540 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60368613 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1871155323742960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚512129562615808 _step‚540 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1871894037030912 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1871894010857472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚11852.126971928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6785678098944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116832926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56455211 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37134278 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚541 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60367631 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116822842 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512208517072896 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6784332655616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116822842 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1878207097935702 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1871434125516736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1871434150944704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19808414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19808070 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1872172920147968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116832926 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37133929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1878958624420352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1878957252803584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8299478485504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1872172946321408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60367631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60377709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56455217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56455217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520507627467264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6774317030364 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 34820761674z¼7¯‘N FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚11852.126971928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6785678098944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116832926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56455211 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37134278 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚541 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60367631 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116822842 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512208517072896 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6784332655616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116822842 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1878207097935702 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1871434125516736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1871434150944704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19808414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19808070 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1872172920147968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116832926 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37133929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1878958624420352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1878957252803584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8299478485504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1872172946321408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60367631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60377709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56455217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56455217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520507627467264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6774317030364 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1872172920147968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚512208148981760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6772972418966 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚6499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1872172946321408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520508053109760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 % _timestamp‚1.7848510887693722e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60377709 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚06"64°/ _wandbruntime‚11852  _runtime‚11852€¶x±‚ 0loss/FBt1/assignment_loss‚0.1332228611878996 2loss/FBt2/assignment_loss‚0.023372174951856325 /loss/FRt2/detection_loss‚1.4254800782045767 epoch‚23 trainer/global_step‚6499 % _timestamp‚1.7848510888032775e+09 0loss/FRt1/assignment_loss‚1.6684550576805477 /loss/FBt1/detection_loss‚0.7081464748720036 /loss/FBt2/detection_loss‚0.5163169149521278 _runtime‚11852.160492054 _step‚542 /loss/FRt2/assignment_loss‚0.755543139245775 /loss/FRt1/detection_loss‚0.5268270909071414 1loss/SRqqt2/detection_loss‚0.6356418620684982 2loss/SRqqt1/assignment_loss‚0.5599401510730311 2loss/SRqqt2/assignment_loss‚0.4989555027573062 1loss/SRqqt1/detection_loss‚0.9557990830090611 %loss/total_loss‚8.407700390909826j>Y“ ²‡ 0loss/FBt1/assignment_loss‚0.1332228611878996 2loss/FBt2/assignment_loss‚0.023372174951856325 /loss/FRt2/detection_loss‚1.4254800782045767 epoch‚23 trainer/global_step‚6499 % _timestamp‚1.7848510888032775e+09 0loss/FRt1/assignment_loss‚1.6684550576805477 /loss/FBt1/detection_loss‚0.7081464748720036 /loss/FBt2/detection_loss‚0.5163169149521278 _runtime‚11852.160492054 _step‚542 /loss/FRt2/assignment_loss‚0.755543139245775 /loss/FRt1/detection_loss‚0.5268270909071414 1loss/SRqqt2/detection_loss‚0.6356418620684982 2loss/SRqqt1/assignment_loss‚0.5599401510730311 2loss/SRqqt2/assignment_loss‚0.4989555027573062 1loss/SRqqt1/detection_loss‚0.9557990830090611 %loss/total_loss‚8.407700390909826žæÿ>П³‚(:” ÕŠÓîô®³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.576' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.85866666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O?y2E´‚(:º  ÕŠÓçïϳ disk.drbd1285.in‚0( proc.memory.availableMB‚ 1944.5703125) disk./.usagePercent‚70.51368227735311 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1550.out‚0 network.recv‚735862 disk.rbd1.in‚0 disk.rbd1.out‚ 61.26171875 disk.sda1.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚37249.58740234375 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 304.90234375 cpu‚25.325825280674074 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 88.57421875) proc.memory.percent‚11.77528699239095 network.sent‚ 1.1615955e+07 disk.nvme0n1.in‚ 88.57421875 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚37249.58740234375 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 304.90234375 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8562545776367 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0$ memory_percent‚92.08752314249674 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0É=“µ‚(:ˆ ¬ÕŠÓ™šÂ³ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.504' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©š@¶‚(:µ  ¬ÕŠÓ¨¨â³ disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.rbd1.out‚ 61.2939453125 proc.cpu.threads‚17 network.recv‚736818 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0% disk./.usageGB‚1260.8561973571777 disk.drbd1289.out‚0# disk.dm-0.out‚37250.29833984375 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.rbd1.in‚0 disk.drbd1285.out‚0 cpu‚25.325729176984947 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.drbd1285.in‚0 network.sent‚ 1.1618993e+07 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1226.out‚0# memory_percent‚92.0867919921875 disk.dm-0.in‚ 88.57421875 disk.drbd1938.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚37250.29833984375 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.sda.out‚ 305.31640625 disk.sda2.out‚ 305.31640625 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0) proc.memory.percent‚11.77528699239095# proc.memory.availableMB‚1944.75( disk./.usagePercent‚70.5136790772855 disk.drbd1552.in‚0 disk.sda1.in‚0,ˆÕŸ·‚(:” »ÕŠÓÆ·ä³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚309.566' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.18866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÁ“µA¸‚(:¶  »ÕŠÓ¶Š…´ disk.drbd2118.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51367139712323 disk.sda1.out‚0 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 305.75390625 disk.drbd1356.in‚0# memory_percent‚92.0867919921875 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0# disk.dm-0.out‚37250.73388671875 disk.drbd2078.in‚0 disk.drbd2078.out‚0 network.recv‚737877 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 network.sent‚ 1.1621968e+07 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 305.75390625 disk.rbd1.out‚ 61.3955078125 disk.sda2.in‚0.03125# proc.memory.availableMB‚1944.75 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1289.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚37250.73388671875 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 cpu‚25.325653699653664 disk.drbd2111.in‚0% disk./.usageGB‚1260.8560600280762 disk.drbd2041.in‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0) proc.memory.percent‚11.77528699239095ç Äߟ¹‚(:” ÊÕŠÓÑ‘¼³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.416' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.47200000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³_o‡Fº‚(:»  ÊÕŠÓ‹¾Ü³ disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 88.57421875 disk.drbd2111.in‚0) disk./.usagePercent‚70.51383588059853 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 88.57421875 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0$ memory_percent‚92.08682378133139 disk.sda2.out‚ 306.26171875 disk.drbd1938.in‚0 disk.rbd1.out‚ 61.6376953125 cpu‚25.325581027930223 disk.drbd1550.out‚0 network.sent‚ 1.1622986e+07 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0( proc.memory.availableMB‚ 1944.7421875$ disk./.usageGB‚1260.859001159668 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.sda1.out‚0 network.recv‚738849 disk.drbd1356.out‚0) proc.memory.percent‚11.77528699239095 proc.cpu.threads‚17 disk.drbd1817.in‚0& disk.nvme0n1.out‚37250.94482421875 disk.drbd1510.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 306.26171875 disk.drbd1673.in‚0# disk.dm-0.out‚37250.94482421875 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1965.out‚0£Lá ž»‚(:“ ÙÕŠÓÀ–ѳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.104' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.70133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âg;C¼‚(:¸  ÙÕŠÓ‡î³ disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1944.9921875 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 88.57421875& disk.nvme0n1.out‚37251.38330078125 disk.drbd1510.out‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚37251.38330078125 cpu‚25.325485613428025 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0) proc.memory.percent‚11.77528699239095% disk./.usageGB‚1260.8574752807617 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 network.sent‚ 1.1627819e+07 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 61.6376953125 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 network.recv‚739179( disk./.usagePercent‚70.5137505454622 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1226.out‚0 disk.sda.out‚ 306.7890625 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1289.in‚0# memory_percent‚92.0858065287272 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 306.7890625 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.dm-0.in‚ 88.57421875ùž*Æ4½/ _wandbruntime‚11937  _runtime‚11937Y¾éǾÁ _step‚543 & lr-AdamW/pg1‚6.646568135135311e-05 & lr-AdamW/pg2‚6.646568135135311e-05 trainer/global_step‚6549 $ _timestamp‚1.784851173919674e+09 _runtime‚11937.276926225)²ÎÌ¿Æ & lr-AdamW/pg1‚6.646568135135311e-05 & lr-AdamW/pg2‚6.646568135135311e-05 trainer/global_step‚6549 $ _timestamp‚1.784851173919674e+09 _runtime‚11937.276926225 _step‚543Ÿ˜)ƒ4À/ _wandbruntime‚11937  _runtime‚119371«‚R(Á‚P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1892673975086080 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1885839475382272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516059249004544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56865200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37408251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56865194 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17455219 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19953402 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8360588460032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6824400111420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1892675346695680 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1885839501555712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6835845139968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60814501 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1892673975086080 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1891919837971148 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚11937.278332951 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8360641824256 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1892675346695680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117689765 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6834499703808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1885839501555712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6823055500134 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516058880913408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56865194 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17455224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1885095412431760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚6549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6835845139968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824ïÈÁë‡P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117679695 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60824565 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6834499703808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19953032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524419890828800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37408626 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚524419469373440 % _timestamp‚1.7848511739207673e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMoniúÅÚ"tor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6834499703808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1885839501555712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6823055500134 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516058880913408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56865194 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17455224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1885095412431760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚6549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6835845139968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1892673975086080 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1885839475382272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516059249004544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56865200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37408251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56865194 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17455219 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19953402 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8360588460032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6824400111420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1892675346695680 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1885839501555712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6835845139968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60814501 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1892673975086080 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1891919837971148 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚11937.278332951 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8360641824256 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1892675346695680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117689765 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56873567 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56873561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37413831 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56873567 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1892955278501376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1886118384672768 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6835522211840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚545 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17457859 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37414204 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6825420913172 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1886118410846208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6836867655168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 _runtime‚11938.753882856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8361895458304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1892198290507310 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60833661 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6835522211840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8361842101248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1892199660546676 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1885374239633504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60823583 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516137467279360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524499730828800 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117707228 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56873561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60823583 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6836867655168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1886118384672768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1885374214205536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117707228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19955972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1886118410846208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60833661 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚373 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117707228 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56873561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60823583 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6836867655168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1886118384672768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1885374214205536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117707228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19955972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1886118410846208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60833661 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚373 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56873567 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56873561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37413831 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56873567 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1892955278501376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1886118384672768 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6835522211840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚545 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17457859 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37414204 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6825420913172 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1886118410846208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6836867655168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 _runtime‚11938.753882856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8361895458304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1892198290507310¡Ž˜<4Æ/ _wandbruntime‚11938  _runtime‚11938ìèÿ> Ç… % _timestamp‚1.7848511754345284e+09 trainer/global_step‚6549 _runtime‚11938.79172994 0loss/FBt2/detection_loss‚0.41387787424559214 0loss/FRt1/assignment_loss‚1.6561985308779958 2loss/SRqqt2/assignment_loss‚0.5074898121431045 2loss/FBt2/assignment_loss‚0.004918683546627946 /loss/FBt1/detection_loss‚0.9123286893043447 _step‚546 0loss/FRt2/assignment_loss‚0.7138722815227099 1loss/SRqqt1/detection_loss‚0.9417844408647351 1loss/SRqqt2/detection_loss‚0.6278422895862997 %loss/total_loss‚8.279448112700788 epoch‚23 2loss/SRqqt1/assignment_loss‚0.5277178545681224 2loss/FBt1/assignment_loss‚0.004319100255703871 /loss/FRt1/detection_loss‚0.5639166908794819 /loss/FRt2/detection_loss‚1.4051818649060703SšÆ¦ÈŠ epoch‚23 2loss/SRqqt1/assignment_loss‚0.5277178545681224 2loss/FBt1/assignment_loss‚0.004319100255703871 /loss/FRt1/detection_loss‚0.5639166908794819 /loss/FRt2/detection_loss‚1.4051818649060703 % _timestamp‚1.7848511754345284e+09 trainer/global_step‚6549 _runtime‚11938.79172994 0loss/FBt2/detection_loss‚0.41387787424559214 0loss/FRt1/assignment_loss‚1.6561985308779958 2loss/SRqqt2/assignment_loss‚0.5074898121431045 2loss/FBt2/assignment_loss‚0.004918683546627946 /loss/FBt1/detection_loss‚0.9123286893043447 _step‚546 0loss/FRt2/assignment_loss‚0.7138722815227099 1loss/SRqqt1/detection_loss‚0.9417844408647351 1loss/SRqqt2/detection_loss‚0.6278422895862997 %loss/total_loss‚8.279448112700788¢;ØôàžÉ‚(:“ èÕŠÓÓ«³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚283.823' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.60766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o§×·>Ê‚(:³  èÕŠÓÛáγ disk.sda2.out‚ 307.21875 disk.drbd1965.in‚0$ memory_percent‚92.08699862162271 cpu‚25.325433888112435 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0) proc.memory.percent‚11.77528699239095 disk.drbd1285.out‚0 disk.rbd1.out‚ 61.7939453125 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.sda2.in‚0.03125) disk./.usagePercent‚70.51380089319264 disk.drbd1510.in‚0) proc.memory.availableMB‚ 1944.69921875 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 network.sent‚ 1.1627927e+07 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 network.recv‚739287 disk.nvme0n1.in‚ 88.57421875 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0! disk.dm-0.out‚37251.791015625 proc.cpu.threads‚17% disk./.usageGB‚1260.8583755493164 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 307.21875 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1285.in‚0 disk.drbd2111.in‚0$ disk.nvme0n1.out‚37251.791015625 disk.sda1.in‚0 disk.dm-0.in‚ 88.57421875 disk.drbd1226.in‚0ò‹TžË‚(:“ ÷ÕŠÓþ‹ª³ gpu.0.gpu‚55.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚294.767' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.25566666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ*wFÌ‚(:»  ÷ÕŠÓÎÈʳ) disk./.usagePercent‚70.51402127118223 disk.drbd1289.out‚0) proc.memory.percent‚11.77528699239095 network.recv‚743743 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 307.44140625 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚37252.61962890625 disk.drbd1673.in‚0 disk.drbd2041.in‚0$ memory_percent‚92.08699862162271 disk.nvme0n1.in‚ 88.76953125 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 61.8955078125 disk.drbd1552.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 307.44140625 disk.dm-0.in‚ 88.76953125 disk.drbd2078.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1944.69921875 cpu‚25.3254022054444 disk.sda.in‚0.03125 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0% disk./.usageGB‚1260.8623161315918 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 network.sent‚ 1.1691342e+07 disk.drbd1550.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚37252.61962890625 disk.drbd1356.in‚0«•Ë=Í‚(:¶  †ÖŠÓ÷Åã³) proc.memory.availableMB‚ 1944.69921875 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.rbd1.out‚ 62.0439453125 disk.drbd1552.in‚0 disk.drbd1931.out‚0 network.recv‚743917% disk./.usageGB‚1260.8622817993164 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 308.109375 disk.drbd2041.out‚0 disk.drbd2111.out‚0 network.sent‚ 1.1692189e5³Y#+07 disk.drbd2078.in‚0 disk.dm-0.in‚ 88.76953125 disk.nvme0n1.in‚ 88.76953125 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚37252.9755859375 disk.drbd1673.in‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2893.89453125) proc.memory.percent‚11.77528699239095$ memory_percent‚92.08699862162271) disk./.usagePercent‚70.51401935114167 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0" disk.dm-0.out‚37252.9755859375 disk.drbd2078.out‚0 cpu‚25.32536959986647 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 308.109375 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17}¬üžÎ‚(:“ †ÖŠÓ¼‹¾³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.951' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.98366666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ8,‘žÏ‚(:“ •ÖŠÓÜÚ˳ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.182' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.06066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹sH EЂ(:º  •ÖŠÓÞ‡é³ disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0# disk.dm-0.out‚37253.49169921875 network.recv‚743983 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 cpu‚25.325296151707807 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8623237609863 disk.nvme0n1.in‚ 89.31640625 proc.cpu.threads‚17$ memory_percent‚92.08564758300781 disk.drbd1938.out‚0) disk./.usagePercent‚70.51402169785791 disk.drbd1817.in‚0 disk.sda.out‚ 310.62109375 disk.sda2.in‚0.03125 disk.drbd1510.in‚0& proc.memory.availableMB‚ 1945.03125# proc.memory.rssMB‚ 2893.89453125 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0) proc.memory.percent‚11.77528699239095 disk.sda2.out‚ 310.62109375 disk.drbd1965.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚37253.49169921875 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 network.sent‚ 1.1692255e+07 disk.drbd1289.in‚0 disk.dm-0.in‚ 89.31640625 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 62.0439453125 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1550.in‚0z„|9CÑ‚(:¸  ¤ÖŠÓÃÁÿ³$ memory_percent‚92.08564758300781 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0) proc.memory.percent‚11.77528699239095 disk.drbd2118.out‚0 disk.sda.out‚ 310.97265625 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚37253.69873046875 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2893.89453125 cpu‚25.325243071953714 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 89.31640625# disk.dm-0.out‚37253.69873046875 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 network.sent‚ 1.1700366e+07 disk.drbd1965.out‚0 disk.rbd1.out‚ 62.076171875 network.recv‚746252( disk./.usagePercent‚70.5140184977903 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8622665405273 disk.drbd1673.out‚0 disk.sda2.out‚ 310.97265625 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0& proc.memory.availableMB‚ 1945.03125 disk.nvme0n1.in‚ 89.31640625ý±O’Ò‚(:‡ ¤ÖŠÓÅá³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚328.782' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚109.594 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã]§…žÓ‚(:“ ³ÖŠÓîžÂ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚280.526' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.50866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 þŠ/CÔ‚(:¸  ³ÖŠÓóŽÝ³ disk.drbd2111.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2893.89453125% disk./.usageGB‚1260.8622856140137 disk.drbd2078.in‚0 cpu‚25.325150586138427 disk.drbd1673.out‚0 disk.dm-0.in‚ 89.31640625 network.sent‚ 1.1700432e+07 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0& disk.nvme0n1.out‚37254.06982421875) proc.memory.percent‚11.77528699239095 network.recv‚746318 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1938.out‚0# disk.dm-0.out‚37254.06982421875 disk.nvme0n1.in‚ 89.31640625$ memory_percent‚92.08564758300781 disk.drbd1550.in‚0 disk.sda.out‚ 311.40234375 disk.drbd1356.in‚0 disk.sda2.out‚ 311.40234375( disk./.usagePercent‚70.5140195644795 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 62.177734375& proc.memory.availableMB‚ 1945.03125 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0] Äo4Õ/ _wandbruntime‚12024  _runtime‚12024Õ{ºZÆÖÀ _step‚547 % lr-AdamW/pg1‚6.58008058944496e-05 % lr-AdamW/pg2‚6.58008058944496e-05 trainer/global_step‚6599 % _timestamp‚1.7848512607859197e+09 _runtime‚12024.143181125?þgË×Å % lr-AdamW/pg2‚6.58008058944496e-05 trainer/global_step‚6599 % _timestamp‚1.7848512607859197e+09 _runtime‚12024.143181125 _step‚547 % lr-AdamW/pg1‚6.58008058944496e-05£ݹ¡Ü4Ø/ _wandbruntime‚12024  _runtime‚12024?ÃÀx(Ù‚P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17587219 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57283550 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17587224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1905911030542756 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1899035501120560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519988199211008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20102940 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1899784966080512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1899784939907072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20102556 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519988567302144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1899035526548528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1899784966080512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚12024.147928318 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61270453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6887034696192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37690164 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1905909660503502 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚6599 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1906670629167104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118564067 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6885689260032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8423096728064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118553997 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1906670629167104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6885689260032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61280517 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1906672000776704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61270453 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6887034696192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚548 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528411664030208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37689775 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61280517 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1906672000776704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8423039169536 % _timestamp‚1.7848512607870662e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1899784966080512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1899784939907072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20102556 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519988567302144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1899035526548528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1899784966080512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚12024.147928318 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61270453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6887034696192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37690164 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1905909660503502 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚6599 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1906670629167104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118564067 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6885689260032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8423096728064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118553997 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1906670629167104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050¤!N±4Û/ _wandbruntime‚12025  _runtime‚12025.¶îûôÜŒN M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚375 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520066785576960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8424282595328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1906190853118284 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17589859 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1899314302894336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17589864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1906951932582400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1900063849197568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57291917 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37695379 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚380 X„·õ[DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528491068172288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6886711768064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61279535 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6886711768064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1906950560965632 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118581530 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57291911 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61289613 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6888057211392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520067153668096 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1900063875371008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61289613 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37695759 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6888057211392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57291917 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528491491717632 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118581530 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1900063849197568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57291917 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37695379 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚380 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528491068172288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6886711768064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61279535 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6886711768064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1906950560965632 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118581530 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57291911 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61289613 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6888057211392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520067153668096 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1900063875371008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61289613 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37695759 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6888057211392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57291917 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528491491717632¥|)iå4Þ/ _wandbruntime‚12025  _runtime‚12025¡Ìé ߆ /loss/FBt2/detection_loss‚0.5023010298278471 % _timestamp‚1.7848512623012934e+09 2loss/SRqqt1/assignment_loss‚0.5368922294286917 3loss/FBt1/assignment_loss‚0.0010085531129671033 /loss/FBt1/detection_loss‚0.5321020096889697 _step‚550 0loss/FRt1/assignment_loss‚1.7624155101343888 2loss/SRqqt2/assignment_loss‚0.5436169862142772 1loss/SRqqt1/detection_loss‚0.9718044743869841 1loss/SRqqt2/detection_loss‚0.6109706303758886 %loss/total_loss‚8.064892706116957 trainer/global_step‚6599 _runtime‚12025.658509363 0loss/FRt2/assignment_loss‚0.6647900738954003 2loss/FBt2/assignment_loss‚0.027002426657261474 /loss/FRt1/detection_loss‚0.5194525149482522 /loss/FRt2/detection_loss‚1.3925362674460289 epoch‚23æÂÓà‹ /loss/FBt2/detection_loss‚0.5023010298278471 % _timestamp‚1.7848512623012934e+09 2loss/SRqqt1/assignment_loss‚0.5368922294286917 3loss/FBt1/assignment_loss‚0.0010085531129671033 /loss/FBt1/detection_loss‚0.5321020096889697 _step‚550 0loss/FRt1/assignment_loss‚1.7624155101343888 2loss/SRqqt2/assignment_loss‚0.5436169862142772 1loss/SRqqt1/detection_loss‚0.9718044743869841 1loss/SRqqt2/detection_loss‚0.6109706303758886 %loss/total_loss‚8.064892706116957 trainer/global_step‚6599 _runtime‚12025.658509363 0loss/FRt2/assignment_loss‚0.6647900738954003 2loss/FBt2/assignment_loss‚0.027002426657261474 /loss/FRt1/detection_loss‚0.5194525149482522 /loss/FRt2/detection_loss‚1.3925362674460289 epoch‚23¦{­¬@á‚(:µ  Â֊Ӿ̳ disk.drbd2111.in‚0) proc.memory.percent‚11.77528699239095 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚70.5140195644795 disk.drbd1285.in‚0 disk.drbd1828.in‚0 network.recv‚749347 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.dm-0.in‚ 89.31640625# disk.dm-0.out‚37254.24951171875 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 311.67578125 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 cpu‚25.325099161077187& disk.nvme0n1.out‚37254.24951171875 disk.drbd2118.out‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8622856140137 disk.drbd1938.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.sda1.out‚0 disk.drbd2118.in‚0# proc.memory.availableMB‚1945.25 disk.sda2.in‚0.03125 disk.rbd1.out‚ 62.240234375 disk.drbd1289.in‚0 disk.drbd1965.out‚0$ memory_percent‚92.08475748697916 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 311.67578125 disk.drbd2078.out‚0 disk.drbd1356.in‚0 network.sent‚ 1.1706877e+07 disk.nvme0n1.in‚ 89.31640625Ý@Råžâ‚(:“ ÂÖŠÓž¢¬³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚272.573' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.85766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 BãŸã‚(:” ÑÖŠÓÑѳ³ gpu.0.gpu‚38.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚304.454' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.48466666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÂËDä‚(:¹  ÑÖŠÓߞس disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 cpu‚25.325026186843537 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚37254.56982421875% disk./.usageGB‚1260.8622856140137 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 network.recv‚751327 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 62.419921875 disk.sda2.out‚ 312.0859375 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 89.31640625 network.sent‚ 1.1766783e+07( disk./.usagePercent‚70.5140195644795 disk.drbd1938.out‚0# disk.dm-0.out‚37254.56982421875 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1944.99609375 disk.dm-0.in‚ 89.31640625 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 312.0859375# proc.memory.rssMB‚ 2893.89453125 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0$ memory_percent‚92.08579063415527 proc.cpu.threads‚17 disk.sda.in‚0.03125 disk.drbd1356.out‚0) proc.memory.percent‚11.77528699239095ó±\Õ“å‚(:ˆ àÖŠÓÒ¿±³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.334' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.778 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;»%÷Gæ‚(:¼  àÖŠÓÁêÓ³ disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0) proc.memory.availableMB‚ 1944.99609375% disk./.usageGB‚1260.8603439331055 disk.drbd1828.in‚0& disk.nvme0n1.out‚37255.25732421875 disk.sda2.out‚ 312.50390625) disk./.usagePercent‚70.51391097551851 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 89.31640625$ memory_percent‚92.08579063415527 disk.sda.in‚0.03125 disk.sda.out‚ 312.50390625 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1938.out‚0) proc.memory.percent‚11.77528699239095 disk.nvme0n1.in‚ 89.31640625 disk.drbd1552.in‚0 network.recv‚751435 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 network.sent‚ 1.1766891e+07 disk.drbd2118.out‚0 cpu‚25.324912614384594 disk.rbd1.out‚ 62.482421875 disk.drbd1226.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚37255.25732421875àNÃCžç‚(:“ ïÖŠÓ ¡µ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.374' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.12466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªìx¶Eè‚(:º  ïÖŠÓÂ׳ disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1944.99609375 disk.drbd1285.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51349966016134 disk.drbd1817.out‚0 disk.drbd1938.in‚0 network.recv‚751501% disk./.usageGB‚1260.8529891967773 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 89.31640625 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0) proc.memory.percent‚11.77528699239095 disk.drbd2118.in‚0 disk.sda.out‚ 312.8046875 disk.dm-0.in‚ 89.31640625 disk.drbd1673.in‚0 disk.sda2.out‚ 312.8046875 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 62.669921875& disk.nvme0n1.out‚37256.41162109375# disk.dm-0.out‚37256.41162109375 network.sent‚ 1.1766957e+07 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1289.in‚0 cpu‚25.324902278592226 disk.drbd1584.in‚0 disk.drbd2111.in‚0$ memory_percent‚92.08579063415527 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1226.out‚0b,ìíEé‚(:º  þÖŠÓôî׳ disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 313.3046875 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1965.in‚0) disk./.usagePercent‚70.51350072685055 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚37256.55615234375 network.sent‚ 1.1775582e+07 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 313.3046875 disk.rbd1.out‚ 62.732421875 network.recv‚753794 disk.sda1.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0& disk.nvme0n1.out‚37256.55615234375) proc.memory.availableMB‚ 1944.99609375 disk.drbd1673.out‚0 disk.drbd1673.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2893.89453125% disk./.usageGB‚1260.8530082702637 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0) proc.memory.percent‚11.77528699239095 disk.drbd1552.out‚0 disk.drbd2111.out‚0$ memory_percent‚92.08579063415527 disk.drbd1226.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.drbd2078.out‚0 cpu‚25.324850819957575 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1226.out‚0%uî“ê‚(:ˆ þÖŠÓ¢×µ³ gpu.0.gpu‚42.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚289.017' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.339 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ÁЂžë‚(:“ ׊Ӄź³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.176' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.05866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç×ÁÜ:ì‚(:¯  ׊Ó´¬Þ³ disk.sda1.in‚0) proc.memory.availableMB‚ 1945.19921875 disk.drbd1673.in‚0 disk.sda2.out‚313.75 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0% disk./.usageGB‚1260.8530082702637 disk.drbd1673.out‚0 disk.dm-0.in‚ 92.38671875 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 network.sent‚ 1.1778101e+07 network.recv‚754944 disk.drbd1931.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2893.89453125 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda.out‚313.75 disk.drbd1817.out‚0# disk.dm-0.out‚37257.06005859375 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0# memory_percent‚92.0849641164144 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51350072685055 disk.sda2.in‚0.03125 disk.drbd1356.out‚0& disk.nvme0n1.out‚37257.06005859375 disk.drbd1550.in‚0 disk.drbd1584.in‚0 cpu‚25.324737516197395 disk.rbd1.out‚ 62.732421875 disk.drbd1285.out‚0 disk.drbd1226.in‚0) proc.memory.percent‚11.77528699239095 disk.drbd1938.out‚0O4ê+4í/ _wandbruntime‚12110  _runtime‚12110cY¸ÆîÀ % lr-AdamW/pg1‚6.51349819251453e-05 % lr-AdamW/pg2‚6.51349819251453e-05 trainer/global_step‚6649 % _timestamp‚1.7848513472397654e+09 _runtime‚12110.597843803 _step‚551…¿ììËïÅ % lr-AdamW/pg2‚6.51349819251453e-05 trainer/global_step‚6649 % _timestamp‚1.7848513472397654e+09 _runtime‚12110.597843803 _step‚551 % lr-AdamW/pg1‚6.51349819251453e-05§4‡{4ð/ _wandbruntime‚12110  _runtime‚12110e"ã¹ ñ‚P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1913730404431872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37971614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57701900 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57701900 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57701894 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚387 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523917517508608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6925263265750 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1920667283248128 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1919900853075110 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6926607877036 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1920668654857728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61726405 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17719224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 _runtime‚12110.598097696 _step‚552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6938224252416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6936878816256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532402996846592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523917885599744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚6649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1920668654857728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17719219 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1913730430605312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1920667283248128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57701894 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20252395 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1913730404431872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1913730430605312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119438369 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8485479337984 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1919902223114364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6936878816256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61736469 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1912975615237328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37972001 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6172640562•¨ (ò‡P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1913730404431872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1913730430605312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119438369 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8485479337984 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1919902223114364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6936878816256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61736469 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1912975615237328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37972001 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61726405 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1913730404431872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37971614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57701900 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57701900 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57701894 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚387 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523917517508608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6925263265750 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1920667283248128 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1919900853075110 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6926607877036 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1920668654857728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61726405 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17719224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 _runtime‚12110.598097696 _step‚552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6938224252416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6936878816256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532402996846592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523917885599744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚6649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1920668654857728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17719219 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1913730430605312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1920667283248128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57701894 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20252395 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112¨‘õƒ4ó/ _wandbruntime‚12112  _runtime‚12112è‚kÏ'ôŒN ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1913254391583136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848513487106824e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1914009313722368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57710267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119445748 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57710267 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119445748 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚389 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1920948586663424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6939246767616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1920180675650526 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20255382 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61745565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17721864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚12112.068282459 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6937901324288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6927628678788 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1920182045689892 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119455832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6926284067390 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61735487 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57710261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61735487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1914009339895808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1914009339895808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6939246767616 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1920948586663424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532482834344960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37977241]{ù;õ‘N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532483257890304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848513487106824e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1914009313722368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57710267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119445748 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57710267 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119445748 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚389 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1920948586663424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6939246767616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1920180675650526 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20255382 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61745565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17721864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚12112.068282459 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6937901324288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \JDeviceStatsMonitor.on_train_batch_eš]‰Ünd/requested_bytes.small_pool.allocated‚ 6927628678788 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1920182045689892 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119455832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6926284067390 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61735487 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57710261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61735487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1914009339895808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1914009339895808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6939246767616 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1920948586663424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532482834344960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37977241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1913254391583136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20402352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1933892045646718 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8547996652032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120302601 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1934665308938752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1927675868956672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚389 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1926915703926128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38253965 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62192421 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536394772802560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58120250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1934665308938752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38253571 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17851219 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚556 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58120250 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848514340162342e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62192421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120312671 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1934663937329152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58120244 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527847203897344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1934663937329152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6988068372480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6976367148558 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6977711759844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112¤!¾ Š‡P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6989413808640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1934663937329152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58120244 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527847203897344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1934663937329152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6988068372480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6976367148558 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6977711759844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62182357 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1927675868956672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MÊïGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17851224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6988068372480 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20402352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1933892045646718 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8547996652032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120302601 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1934665308938752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1927675868956672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚389 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1926915703926128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38253965 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62192421 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536394772802560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58120250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1934665308938752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38253571 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17851219 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚556 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58120250 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848514340162342e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62192421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120312671 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6989090880512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚527925790263296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38259167 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120320050 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1927194480271936 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1927954804420608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58128617 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1927954778247168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1934173238261500 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58128611 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17853864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120330134 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62191439 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120330134 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚383 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58128611 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62201517 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20405691 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120320050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1934171868222134 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6978732561596 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1934945240744448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6989090880512 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1927954778247168 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38259555 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0W@”¿'‘N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6977387950198 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6989090880512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚527925790263296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38259167 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120320050 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1927194480271936 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1927954804420608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58128617 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1927954778247168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1934173238261500 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58128611 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17853864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120330134 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62191439 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120330134 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚383 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58128611 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62201517 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20405691 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120320050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1934171868222134 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6978732561596 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1934945240744448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6989090880512 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1927954778247168 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38259555 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚388­¿#qß4Ž/ _wandbruntime‚12198  _runtime‚12198ŹBô ƒ epoch‚23 $ _timestamp‚1.784851435528389e+09 _runtime‚12198.885570815 2loss/SRqqt2/assignment_loss‚0.4958496645115271 1loss/SRqqt2/detection_loss‚0.5897169370031778 0loss/FBt2/detection_loss‚0.47578655725653124 _step‚558 1loss/FBt2/assignment_loss‚0.06011354615585028 /loss/FRt1/detection_loss‚0.5608508102890694 trainer/global_step‚6699 0loss/FRt1/assignment_loss‚1.7192097330684317 0loss/FRt2/assignment_loss‚0.7238149642944336 0loss/SRqqt1/detection_loss‚0.936023394953938 2loss/SRqqt1/assignment_loss‚0.5455031748508898 3loss/FBt1/assignment_loss‚0.0003157264053701425 /loss/FRt2/detection_loss‚1.4425328970480291 .loss/FBt1/detection_loss‚0.847961999475956 %loss/total_loss‚8.397679405313204h^ˆ 2loss/SRqqt1/assignment_loss‚0.5455031748508898 3loss/FBt1/assignment_loss‚0.0003157264053701425 /loss/FRt2/detection_loss‚1.4425328970480291 .loss/FBt1/detection_loss‚0.847961999475956 %loss/total_loss‚8.397679405313204 epoch‚23 $ _timestamp‚1.784851435528389e+09 _runtime‚12198.885570815 2loss/SRqqt2/assignment_loss‚0.4958496645115271 1loss/SRqqt2/detection_loss‚0.5897169370031778 0loss/FBt2/detection_loss‚0.47578655725653124 _step‚558 1loss/FBt2/assignment_loss‚0.06011354615585028 /loss/FRt1/detection_loss‚0.5608508102890694 trainer/global_step‚6699 0loss/FRt1/assignment_loss‚1.7192097330684317 0loss/FRt2/assignment_loss‚0.7238149642944336 0loss/SRqqt1/detection_loss‚0.936023394953938®[À”~ž‘‚(:“ ö׊Óµ—®³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.499' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.16633333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07›8&7’‚(:¬  ö׊ÓÅáÓ³ disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 63.296875 disk.drbd1552.out‚0 disk.drbd2078.out‚0) proc.memory.percent‚11.77528699239095 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8531494140625 disk.drbd1289.in‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0# disk.dm-0.out‚37259.21630859375) disk./.usagePercent‚70.51350862035068 disk.sda.out‚316.1875 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2893.89453125 cpu‚25.324280698994638 disk.drbd2111.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚37259.21630859375 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1226.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.08628336588541 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚316.1875 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 network.sent‚ 1.1848526e+07 disk.drbd1828.in‚0 network.recv‚761844 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1817.out‚0$ proc.memory.availableMB‚1944.875»P6ž“‚(:“ …ØŠÓò…à³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚291.148' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.04933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç)™””‚(:·  …ØŠÓÙ•ü³ disk.drbd1550.in‚0 network.recv‚761994 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2893.89453125 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚37259.39599609375 disk.drbd1510.out‚0) disk./.usagePercent‚70.51346296605273 disk.dm-0.in‚ 92.38671875 disk.drbd1356.in‚0) proc.memory.percent‚11.77528699239095 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 92.38671875 network.sent‚ 1.1848676e+07 disk.drbd1356.out‚0 disk.rbd1.in‚0$ memory_percent‚92.08712577819824 disk.drbd2118.out‚0 disk.sda2.in‚0.03125) proc.memory.availableMB‚ 1944.66796875 disk.drbd2078.in‚0 cpu‚25.3241884449947 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0# disk.dm-0.out‚37259.39599609375 disk.drbd1828˜ý6[®.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 316.66796875 disk.drbd2111.in‚0 disk.sda2.out‚ 316.66796875 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0% disk./.usageGB‚1260.8523330688477 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 63.296875 disk.drbd1226.in‚03¸ò A•‚(:¶  ”ØŠÓËæ³ disk.drbd1285.out‚0# proc.memory.rssMB‚ 2896.19921875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.784664789835613 disk.drbd2118.out‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚37260.35986328125 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 cpu‚25.325239161125843 disk.sda2.out‚ 316.94921875 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 92.38671875 network.recv‚764743 disk.drbd2111.in‚0 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0# memory_percent‚92.0981248219808 disk.sda.out‚ 316.94921875 disk.drbd1356.out‚0 disk.drbd1510.in‚0 network.sent‚ 1.1913858e+07) disk./.usagePercent‚70.51347448629613 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 63.453125& disk.nvme0n1.out‚37260.35986328125 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1941.96484375" disk./.usageGB‚1260.8525390625 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚05#HŒ“–‚(:ˆ ”ØŠÓ—ÔÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚297.216' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.072 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}34—/ _wandbruntime‚12253  _runtime‚12253{$ûe;˜ßv :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06705190241336823 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39325258135795593 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6139045357704163 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 *jet/accuracy_2_of_4‚0.8214783668518066 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9163104295730591 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.312991201877594 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7104636430740356 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6653212308883667 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7229309678077698 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2481519877910614 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7287370562553406 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25613999366760254 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8279633522033691 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8124381899833679 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.040204085409641266 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7555652260780334 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2693673074245453 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690917015075684 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7validation_average_jet_accuracy‚0.39946094155311584 % _timestamp‚1.7848514896900246e+09 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7020059823989868 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7492967844009399 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _runtime‚12253.048115371 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3271007835865021 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6608375906944275 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7507216334342957 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7889845967292786 *jet/accuracy_1_of_3‚0.9401236772537231 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20477215945720673 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16431914269924164 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974471926689148 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6653212308883667 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1327718198299408 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9928030371665955 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6635714769363403 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8306201696395874 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025979042053222656 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25906258821487427 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23811006546020508 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15534932911396027 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8136698603630066 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30472880601882935 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7504591941833496 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5998824238777161 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49242427945137024 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8318259119987488 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925484657287598 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.750099241733551 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7622004151344299 particle/accuracy_1_of_2‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4597046375274658 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8684824705123901 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9985119104385376 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 _step‚559 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7609614133834839 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24763251841068268 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6478022336959839 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6733592748641968 trainer/global_step‚6719 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14186765253543854 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8705893754959106 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3949543535709381 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027569005265831947 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2710970342159271 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 /particle/accuracy_4_of_4‚0.9796996712684631 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8766719102859497 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12412815541028976 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7514427304267883 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6589214205741882 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4730836749076843 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3371853232383728 *particle/specificity‚0.906783938407898 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.647367537021637 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6887518763542175 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_1_of_2‚0.8033362627029419 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25590234994888306 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.81473308801651 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24286185204982758 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8285946846008301 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7908833622932434 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32538464665412903 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03323379531502724 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.240012814179872 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3350751996040344 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6147481203079224 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 *jet/accuracy_3_of_4‚0.1148163229227066 (particle/accuracy‚0.8517194390296936 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9915838241577148 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7589210867881775 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.705807089805603 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9969428181648254 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05064310505986214 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46626749634742737 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03539709001779556 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.678146481513977 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.714271068572998 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23277008533477783 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2557103931903839 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.368662029504776 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4423076808452606 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.30380263924598694 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13741478323936462 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7549915313720703 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5215193033218384 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7943611741065979 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2963370084762573 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6183182001113892 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46396371722221375 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.313524067401886 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 *jet/accuracy_3_of_3‚0.0730024203658104 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.313524067401886 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36649391055107117 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9411315321922302 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.699026882648468 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1008351594209671 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7370625734329224 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 +particle/sensitivity‚0.6501263976097107 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5876665711402893 *jet/accuracy_1_of_1‚0.3626851439476013 'particle/f_score‚0.6529421806335449 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6682432293891907 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047984056174755096 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8684824705123901 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20161230862140656 *jet/accuracy_1_of_4‚0.9707711338996887 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6464033722877502 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4467781186103821 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32599106431007385 *jet/accuracy_2_of_3‚0.5892836451530457 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7"Purity/2FRt0SRqqt1FBt/event_purity‚0.54358971118927 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 epoch‚23 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.99146568775177µ%£»:™äv 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4467781186103821 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32599106431007385 *jet/accuracy_2_of_3‚0.5892836451530457 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7"Purity/2FRt0SRqqt1FBt/event_purity‚0.54358971118927 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 epoch‚23 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.99146568775177 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06705190241336823 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39325258135795593 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6139045357704163 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 *jet/accuracy_2_of_4‚0.8214783668518066 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9163104295730591 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.312991201877594 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7104636430740356 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6653212308883667 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7229309678077698 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2481519877910614 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7287370562553406 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25613999366760254 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8279633522033691 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8124381899833679 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.040204085409641266 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7555652260780334 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2693673074245453 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690917015075684 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7validation_average_jet_accuracy‚0.39946094155311584 % _timestamp‚1.7848514896900246e+09 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7020059823989868 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7492967844009399 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _runtime‚12253.048115371 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3271007835865021 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6608375906944275 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7507216334342957 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7889845967292786 *jet/accuracy_1_of_3‚0.9401236772537231 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20477215945720673 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16431914269924164 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974471926689148 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6653212308883667 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1327718198299408 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9928030371665955 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6635714769363403 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8306201696395874 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025979042053222656 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25906258821487427 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23811006546020508 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15534932911396027 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8136698603630066 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30472880601882935 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7504591941833496 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5998824238777161 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49242427945137024 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8318259119987488 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925484657287598 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.750099241733551 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7622004151344299 particle/accuracy_1_of_2‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4597046375274658 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8684824705123901 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9985119104385376 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967532157897949 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 _step‚559 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7609614133834839 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24763251841068268 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6478022336959839 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6733592748641968 trainer/global_step‚6719 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14186765253543854 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8705893754959106 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3949543535709381 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027569005265831947 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2710970342159271 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 /particle/accuracy_4_of_4‚0.9796996712684631 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8766719102859497 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12412815541028976 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7514427304267883 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6589214205741882 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4730836749076843 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3371853232383728 *particle/specificity‚0.906783938407898 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.647367537021637 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6887518763542175 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_1_of_2‚0.8033362627029419 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25590234994888306 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.81473308801651 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24286185204982758 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8285946846008301 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7908833622932434 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32538464665412903 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03323379531502724 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.240012814179872 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3350751996040344 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6147481203079224 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 *jet/accuracy_3_of_4‚0.1148163229227066 (particle/accuracy‚0.8517194390296936 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9915838241577148 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7589210867881775 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.705807089805603 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952381253242493 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9969428181648254 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05064310505986214 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9166666269302368 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46626749634742737 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03539709001779556 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.678146481513977 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.714271068572998 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23277008533477783 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2557103931903839 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.368662029504776 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4423076808452606 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.30380263924598694 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13741478323936462 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7549915313720703 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5215193033218384 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7943611741065979 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2963370084762573 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6183182001113892 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46396371722221375 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.313524067401886 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 *jet/accuracy_3_of_3‚0.0730024203658104 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.313524067401886 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36649391055107117 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9411315321922302 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.699026882648468 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1008351594209671 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7370625734329224 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 +particle/sensitivity‚0.6501263976097107 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5876665711402893 *jet/accuracy_1_of_1‚0.3626851439476013 'particle/f_score‚0.6529421806335449 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6682432293891907 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047984056174755096 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8684824705123901 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20161230862140656 *jet/accuracy_1_of_4‚0.9707711338996887 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purit«¯þ³¯y/0FRt*SRqqt0FBt/event_purity‚0.6464033722877502 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936¯tX,Fš‚(:’ £ØŠÓæó¶³ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚57.0 gpu.0.powerWatts‚105.701' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.233666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h:™B›‚(:·  £ØŠÓ åÓ³ disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0% disk./.usageGB‚1262.0541191101074 disk.drbd2111.in‚0 disk.drbd1938.in‚0 disk.drbd1550.in‚0) proc.memory.percent‚15.73030153910319 disk.sda2.out‚ 317.32421875# disk.dm-0.out‚37260.71923828125 disk.drbd1550.out‚0& disk.nvme0n1.out‚37260.71923828125 disk.drbd1510.out‚0 disk.sda.out‚ 317.32421875 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 cpu‚25.327914740410137) disk./.usagePercent‚70.58067313277164 disk.sda.in‚0.03125 disk.drbd2111.out‚0 disk.rbd1.out‚ 63.5546875 network.sent‚ 1.1913924e+07 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0$ memory_percent‚99.98151461283365 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 network.recv‚764809 disk.drbd1817.out‚0 disk.dm-0.in‚ 92.38671875 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0& proc.memory.availableMB‚ 4.54296875 disk.drbd1356.in‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 3865.878906252®¡žœ‚(:“ ²ØŠÓ©Öá³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚293.405' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.80166666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÚ›>‚(:³  ²ØŠÓôñ‚´ disk.drbd1510.out‚0 disk.sda.out‚ 317.6171875 disk.sda2.in‚0.03125 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.51374393198913 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2078.out‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 317.6171875& disk.nvme0n1.out‚38231.14990234375 disk.drbd2118.in‚0$ memory_percent‚88.14020156860352 disk.rbd1.out‚ 63.6171875 disk.nvme0n1.in‚ 92.38671875 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.sda1.in‚0# disk.dm-0.out‚38231.14990234375 disk.rbd1.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1356.in‚0 cpu‚25.32688348734818 network.recv‚768544 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.785252888997395 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1965.in‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 2914.6640625 network.sent‚ 1.197617e+07 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 proc.memory.rssMB‚ 2896.34375% disk./.usageGB‚1260.8573570251465 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0Õýª®“ž‚(:ˆ ÁØŠÓÛ±³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚233.406' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚77.802 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xlBŸ‚(:·  ÁØŠÓçòÓ³ disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 318.16796875& disk.nvme0n1.out‚38231.46630859375 disk.sda.out‚ 318.16796875 disk.drbd1931.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.785252888997395 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 63.703125 disk.sda1.in‚0$ memory_percent‚88.14021746317545 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 network.recv‚768610 disk.dm-0.in‚ 92.38671875 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8573150634766 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 cpu‚25.326832030833053 network.sent‚ 1.1976236e+07 disk.drbd2041.out‚0 disk.sda2.in‚0.03125) disk./.usagePercent‚70.51374158527287 disk.drbd1552.in‚0) proc.memory.availableMB‚ 2914.66015625 proc.memory.rssMB‚ 2896.34375# disk.dm-0.out‚38231.46630859375 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0ÂÞ6ž ‚(:“ ÐØŠÓ‹Ó¼³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚110.782' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.92733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i”úÆ<¡‚(:±  ÐØŠÓ†¦Ý³ disk.drbd1550.in‚0 disk.nvme0n1.in‚ 92.38671875# disk.dm-0.out‚39201.94189453125 disk.drbd1938.out‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚39201.94189453125 disk.dm-0.in‚ 92.38671875 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 cpu‚25.32675855250337 disk.drbd1510.out‚0 network.sent‚ 1.1976344e+07 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51352270064817 disk.drbd1552.out‚0 disk.drbd1289.out‚0$ disk./.usageGB‚1260.853401184082 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 63.765625 disk.drbd1285.in‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1552.in‚0 network.recv‚768718 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 318.44140625 disk.drbd1356.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0& proc.memory.availableMB‚ 2914.65625 disk.drbd1226.out‚0 disk.drbd1673.out‚0# memory_percent‚88.1402333577474 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 318.44140625 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.785252888997395 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0TŸy4¢/ _wandbruntime‚12308  _runtime‚12308L{AýÇ£Á & lr-AdamW/pg1‚6.380083765929313e-05 & lr-AdamW/pg2‚6.380083765929313e-05 trainer/global_step‚6749 $ _timestamp‚1.784851545222841e+09 _runtime‚12308.580075366 _step‚560WešÌ¤Æ & lr-AdamW/pg2‚6.380083765929313e-05 trainer/global_step‚6749 $ _timestamp‚1.784851545222841e+09 _runtime‚12308.580075366 _step‚560 & lr-AdamW/pg1‚6.380083765929313e-05°nOR4¥/ _wandbruntime‚12308  _runtime‚12308”| þ(¦€P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1944182036379648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1951228258183680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8617753371136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58628032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7046221804032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20577785 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7044876367872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1943414793766096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7034424481966 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1951228258183680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531825153616896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚12308.581335579 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1944182010206208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121323717 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62695685 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1944182036379648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17990111 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7044876367872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7046221804032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1951226886574080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17990116 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1943414768338128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚561 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1951226886574080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚540442906988032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864~OW- (§…P ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1943414793766096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7034424481966 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1951228258183680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531825153616896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚12308.581335579 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1944182010206208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121323717 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62695685 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1944182036379648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17990111 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7044876367872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7046221804032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1951226886574080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17990116 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1943414768338128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚561 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1951226886574080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚540442906988032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1944182036379648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1951228258183680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8617753371136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58628032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7046221804032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20577785 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7044876367872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7034100672320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58636393 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚328 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8619054216192 % _timestamp‚1.7848515466999784e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62704781 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62694703 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58636399 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1951508189989376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38573471 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ¦Ø}ÜâXDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540522745036288 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1943693595539872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121331096 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1944460919496704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58636393 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58636399 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1950729040823590 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1944460945670144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7047244319232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚6749 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1944460945670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62694703 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38573799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540523162290176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1944460919496704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7045898875904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121341180 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38573471 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540522745036288 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1943693595539872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121331096 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1944460919496704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58636393 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58636399 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1950729040823590 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1944460945670144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7047244319232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚6749 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1944460945670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62694703 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38573799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540523162290176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1944460919496704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7045898875904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121341180 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7034100672320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58636393 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚328 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8619054216192 % _timestamp‚1.7848515466999784e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328²ß;4«/ _wandbruntime‚12310  _runtime‚12310Æaù ¬„ % _timestamp‚1.7848515467352192e+09 0loss/FRt1/assignment_loss‚1.6703546759672463 0loss/FRt2/assignment_loss‚0.7165451255914207 2loss/SRqqt1/assignment_loss‚0.5289847986161699 2loss/SRqqt2/assignment_loss‚0.5072072222101677 2loss/FBt1/assignment_loss‚0.034408215937309895 2loss/FBt2/assignment_loss‚0.003715457551295762 1loss/SRqqt1/detection_loss‚0.8683873503864902 epoch‚24 /loss/FRt2/detection_loss‚1.4375902811711585 /loss/FBt2/detection_loss‚0.3953611743324038 %loss/total_loss‚7.866464760599587 _step‚563 /loss/FRt1/detection_loss‚0.4994698564398661 _runtime‚12310.092402257 0loss/SRqqt2/detection_loss‚0.622562714387302 /loss/FBt1/detection_loss‚0.5818778880087561 trainer/global_step‚6749öˆ¾­‰ /loss/FRt1/detection_loss‚0.4994698564398661 _runtime‚12310.092402257 0loss/SRqqt2/detection_loss‚0.622562714387302 /loss/FBt1/detection_loss‚0.5818778880087561 trainer/global_step‚6749 % _timestamp‚1.7848515467352192e+09 0loss/FRt1/assignment_loss‚1.6703546759672463 0loss/FRt2/assignment_loss‚0.7165451255914207 2loss/SRqqt1/assignment_loss‚0.5289847986161699 2loss/SRqqt2/assignment_loss‚0.5072072222101677 2loss/FBt1/assignment_loss‚0.034408215937309895 2loss/FBt2/assignment_loss‚0.003715457551295762 1loss/SRqqt1/detection_loss‚0.8683873503864902 epoch‚24 /loss/FRt2/detection_loss‚1.4375902811711585 /loss/FBt2/detection_loss‚0.3953611743324038 %loss/total_loss‚7.866464760599587 _step‚563³_z©®‚(:’ ßØŠÓØïÓ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚289.03' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.34333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~•!B¯‚(:·  ßØŠÓÌôð³ disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 2914.62890625 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 proc.memory.rssMB‚ 2896.34375 disk.sda2.in‚0.03125 disk.sda2.out‚ 318.87890625 disk.dm-0.in‚ 92.38671875 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 network.sent‚ 1.197641e+07 disk.drbd1289.out‚0# disk.dm-0.out‚39202.16845703125 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.rbd1.out‚ 63.8671875 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚39202.16845703125 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8534774780273$ memory_percent‚88.14034461975098 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 92.38671875 disk.sda.out‚ 318.87890625* proc.memory.percent‚11.785252888997395 network.recv‚768784 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 cpu‚25.326725376313508 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.51352696740499Û,Rž°‚(:“ î؊ӟܼ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚252.829' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.27633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Åºd9±‚(:®  îØŠÓ…øܳ disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.drbd1550.out‚0) disk./.usagePercent‚70.51363598304165 disk.drbd1226.out‚0% proc.memory.availableMB‚ 2914.5625 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.sda1.in‚0# disk.dm-0.out‚39202.56005859375 disk.rbd1.out‚ 63.9296875* proc.memory.percent‚11.785252888997395 disk.drbd1226.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2896.34375 network.sent‚ 1.2033989e+07& disk.nvme0n1.out‚39202.56005859375 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.sda.out‚ 319.1953125 disk.dm-0.in‚ 92.38671875 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd2118.in‚0 disk.drbd1938.in‚0 network.recv‚772298# disk./.usageGB‚1260.85542678833 disk.drbd1552.out‚0 disk.sda2.out‚ 319.1953125 cpu‚25.3266329712496 disk.drbd2118.out‚0 disk.drbd2078.out‚0$ memory_percent‚88.14061482747397z  П²‚(:” ý؊Өೠgpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚112.359' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.452999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿHö5³‚(:ª  ýØŠÓΧå³ disk.drbd1226.in‚0% disk./.usageGB‚1260.8554000854492 disk.sda1.out‚0 disk.sda.out‚319.5 disk.drbd1673.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 2914.26171875 disk.sda2.out‚319.5 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 network.recv‚777254 disk.drbd1965.in‚0 cpu‚25.326580215115193 disk.drbd1817.out‚0 disk.rbd1.out‚ 63.9296875 disk.drbd1285.in‚0* proc.memory.percent‚11.785252888997395 network.sent‚ 1.2104744e+07 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚39202.73974609375 disk.drbd1550.out‚0 disk.drbd2111.out‚0# disk.dm-0.out‚39202.73974609375 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1226.out‚0 proc.memory.rssMB‚ 2896.34375$ memory_percent‚88.14183870951334 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51363448967678 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 92.38671875 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd2111.in‚0Vúž´‚(:“ ŒÙŠÓ˜ýä³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.878' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.62599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ø¡ 1µ‚(:¦  ŒÙŠÓ–݃´ disk.sda1.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0$ disk./.usageGB‚1260.855381011963 disk.nvme0n1.in‚ 92.38671875 disk.sda2.in‚0.03125 disk.drbd1938.out‚0) disk./.usagePercent‚70.51363342298757 disk.sda2.out‚319.75 disk.drbd1356.in‚0 disk.drbd2078.out‚0 network.recv‚777386 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 64.171875 disk.drbd1285.out‚0 cpu‚25.32650670109854 disk.drbd2118.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚39203.17138671875 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0' proc.memory.availableMB‚ 2914.265625 disk.drbd1510.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0# memory_percent‚88.1418228149414 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.out‚319.75 disk.drbd1938.in‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 network.sent‚ 1.2105554e+07 disk.drbd1817.out‚0# disk.dm-0.out‚39203.17138671875 disk.drbd1965.in‚0 disk.drbd2041.out‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.785252888997395 disk.sda1.out‚0 disk.drbd1673.out‚0I’l?¶‚(:´  ›ÙŠÓÏÊÙ³ disk.drbd1356.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚39203.36669921875 disk.drbd1584.out‚0 proc.memory.rssMB‚ 2896.34375* proc.memory.percent‚11.785252888997395 disk.rbd1.out‚ 64.2734375 disk.nvme0n1.in‚ 92.38671875 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0' proc.memory.availableMB‚ 2914.265625 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 cpu‚25.32643575623546# memory_percent‚88.1418228149414 disk.rbd1.in‚0 disk.sda.out‚ 320.22265625 disk.drbd1673.out‚0# disk.dm-0.out‚39203.36669921875 network.recv‚778477% disk./.usageGB‚1260.8554191589355 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51363555636597 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.sda.in‚0.03125 disk.drbd1817.in‚0 proc.cpu.threads‚17 network.sent‚ 1.2110597e+07 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 320.22265625 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0Øݲ“·‚(:ˆ ›ÙŠÓƒÚ¸³ gpu.0.gpu‚46.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.368' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.456 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ŒôN:¸‚(:¯  ªÙŠÓðî…´ disk.sda.out‚ 320.57421875 disk.drbd1938.in‚0 disk.drbd1510.out‚0 network.recv‚778585 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 92.38671875) disk./.usagePercent‚70.51363448967678 disk.drbd1550.out‚0* proc.memory.percent‚11.785252888997395 disk.rbd1.out‚ 64.3359375$ disk.nvme0n1.out‚39203.724609375 disk.sda2.in‚0.03125' proc.memory.availableMB‚ 2914.265625 proc.cpu.threads‚17 proc.memory.rssMB‚ 2896.34375 disk.drbd1510.in‚0 network.sent‚ 1.2110705e+07 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8554000854492 disk.drbd2111.out‚0! disk.dm-0.out‚39203.724609375 disk.sda1.out‚0 cpu‚25.3263419906636 disk.sda.in‚0.03125 disk.drbd1931.in‚0# memory_percent‚88.1418228149414 disk.drbd1285.out‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 320.57421875 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0Ï2”ž¹‚(:“ ªÙŠÓ͸æ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚270.011' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.00366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G4º/ _wandbruntime‚12395  _runtime‚12395áðzÆ»À % lr-AdamW/pg2‚6.31326922384918e-05 trainer/global_step‚6799 % _timestamp‚1.7848516320240417e+09 _runtime‚12395.381301421 _step‚564 % lr-AdamW/pg1‚6.31326922384918e-05{6:˼Š% lr-AdamW/pg2‚6.31326922384918e-05 trainer/global_step‚6799 % _timestamp‚1.7848516320240417e+09 _runtime‚12395.381301421 _step‚564 % lr-AdamW/pg1‚6.31326922384918e-05´/Æ-\4½/ _wandbruntime‚12395  _runtime‚12395\‡áB$¾€P _step‚565 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1965224912264704 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59046376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122198019 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18122116 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7096065924096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1964440410819670 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1958127474731008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63151637 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63141573 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535754471914496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 KADeviceStatsMonitor.oßö¦â n_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38847109 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1958127500904448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18122111 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7084183753488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122187949 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1958127474731008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8680273838080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚12395.382590197 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1965223540655104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20724993 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20724645 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59046376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78485163202514e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1965223540655104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38846756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535754840005632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8680335590912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1965224912264704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚6799ØÊ2ì (¿…P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20724993 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20724645 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59046376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78485163202514e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1965223540655104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38846756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535754840005632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8680335590912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1965224912264704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚6799 _step‚565 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1965224912264704 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59046376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122198019 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18122116 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7096065924096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1964440410819670 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1958127474731008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63151637 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63141573 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535754471914496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38847109 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1958127500904448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18122111 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7084183753488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122187949 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1958127474731008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8680273838080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚12395.382590197 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1965223540655104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0µ¢“d4À/ _wandbruntime‚12396  _runtime‚12396ûÔ5Â'Á‹N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122215482 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1965504844070400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20727935 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1965503472453632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59054749 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7097088432128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1957633658800704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63160733 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63150655 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1957633684228672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848516335001607e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18124756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535833426371584 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59054743 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7098433875456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63150655 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1958406410194944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59054743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38852342 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚6799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535833058280448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1964718863355832 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544514998248960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7085204555128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59054749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚566 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122215482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8681571877376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63150655 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1958406410194944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59054743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38852342 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚6799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535833058280448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1964718863355832 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544514998248960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7085204555128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59054749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚566 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122215482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8681571877376 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7097088432128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1957633658800704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¶ã°¡á4Ã/ _wandbruntime‚12396  _runtime‚12396EÿcÄ 1loss/SRqqt1/assignment_loss‚0.564794505577811 2loss/SRqqt2/assignment_loss‚0.5006550596272156 1loss/SRqqt2/detection_loss‚0.6758269001445103 trainer/global_step‚6799 0loss/FRt1/assignment_loss‚1.6576249548741553 0loss/FRt2/assignment_loss‚0.7653692843461478 0loss/SRqqt1/detection_loss‚0.957392457921563 /loss/FBt1/detection_loss‚0.7775525518602902 %loss/total_loss‚8.444650411531747 epoch‚24 1loss/FBt2/assignment_loss‚0.01659979486964073 /loss/FRt1/detection_loss‚0.5464484028555838 /loss/FRt2/detection_loss‚1.4921074126657374 /loss/FBt2/detection_loss‚0.4900083513183297 % _timestamp‚1.7848516335379288e+09 _runtime‚ 12396.8951302 3loss/FBt1/assignment_loss‚0.0002707354707626927 _step‚567sÞãV ņ 1loss/SRqqt1/assignment_loss‚0.564794505577811 2loss/SRqqt2/assignment_loss‚0.5006550596272156 1loss/SRqqt2/detection_loss‚0.6758269001445103 trainer/global_step‚6799 0loss/FRt1/assignment_loss‚1.6576249548741553 0loss/FRt2/assignment_loss‚0.7653692843461478 0loss/SRqqt1/detection_loss‚0.957392457921563 /loss/FBt1/detection_loss‚0.7775525518602902 %loss/total_loss‚8.444650411531747 epoch‚24 1loss/FBt2/assignment_loss‚0.01659979486964073 /loss/FRt1/detection_loss‚0.5464484028555838 /loss/FRt2/detection_loss‚1.4921074126657374 /loss/FBt2/detection_loss‚0.4900083513183297 % _timestamp‚1.7848516335379288e+09 _runtime‚ 12396.8951302 3loss/FBt1/assignment_loss‚0.0002707354707626927 _step‚567·6åXsžÆ‚(:“ ¹ÙŠÓ²¨³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.618' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.87266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒçp+5Ç‚(:ª  ¹ÙŠÓž»Ç³ disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.sda.in‚0.03125 proc.memory.rssMB‚ 2896.34375 cpu‚25.326272105357653 disk.drbd1938.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd2078.in‚0 disk.rbd1.out‚ 64.3359375 disk.drbd1550.out‚0% disk./.usageGB‚1260.8554000854492 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 320.84375 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0$ disk.nvme0n1.out‚39203.958984375 disk.drbd1817.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.785252888997395 disk.drbd1938.out‚0 network.sent‚ 1.2110771e+07 disk.drbd1552.out‚0 disk.drbd1510.out‚0! disk.dm-0.out‚39203.958984375 network.recv‚778651 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51363448967678 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda.out‚ 320.84375 disk.drbd1931.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.09721883138022 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd2111.in‚0% proc.memory.availableMB‚ 1942.1875 ­>aŸÈ‚(:” ÈÙŠÓË̳ gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.673' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.55766666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’¤h>É‚(:³  ÈÙŠÓ´Ÿí³ disk.drbd1817.in‚0 disk.sda.out‚ 321.203125 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 321.203125 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 64.3681640625% disk.nvme0n1.out‚39205.3427734375 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 92.38671875 disk.nvme0n1.in‚ 92.38671875 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1289.in‚0 cpu‚25.326219125949017 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1550.in‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1942.18359375 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0$ memory_percent‚92.09723472595215 disk.drbd1510.out‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.785252888997395 disk.drbd2078.out‚0 disk.drbd1828.in‚0 network.recv‚782449$ disk./.usageGB‚1260.855869293213 disk.drbd1673.in‚0 disk.drbd1289.out‚0 network.sent‚ 1.2176503e+07 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0" disk.dm-0.out‚39205.3427734375( disk./.usagePercent‚70.5136607302312 disk.drbd1226.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0¡fÊ‚(:’ ×ÙŠÓÓ°³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚263.71' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.90333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐU‡@Ë‚(:µ  ×ÙŠÓËÓ³ disk.drbd1552.in‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1942.18359375 disk.drbd1226.in‚0 disk.sda2.out‚ 321.484375 disk.drbd2078.in‚0 disk.drbd2041.out‚0 proc.memory.rssMB‚ 2896.34375% disk.nvme0n1.out‚39205.6435546875 disk.drbd1552.out‚0 disk.drbd1356.out‚0 network.sent‚ 1.2176569e+07 disk.drbd1550.in‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.785252888997395 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 network.recv‚782515 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 cpu‚25.326128288673303) disk./.usagePercent‚70.51366201025824" disk.dm-0.out‚39205.6435546875$ memory_percent‚92.09723472595215 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 64.4697265625 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 321.484375 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125% disk./.usageGB‚1260.8558921813965 disk.drbd1285.out‚0 disk.drbd1226.out‚0<µ²ïžÌ‚(:“ æÙŠÓ œá³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚272.614' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.87133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 9¥H?Í‚(:´  æÙŠÓ¤Üÿ³ disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1931.out‚0 disk.drbd2078.out‚0$ memory_percent‚92.09725062052408% disk./.usageGB‚1260.8558921813965 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.785252888997395 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.dm-0.in‚ 92.38671875 disk.sda.in‚0.03125 network.sent‚ 1.2182832e+07 disk.drbd1285.in‚0$ disk.nvme0n1.out‚39206.041015625 disk.drbd1356.in‚0 disk.sda.out‚ 321.8671875 disk.sda2.out‚ 321.8671875 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 network.recv‚785499 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0! disk.dm-0.out‚39206.041015625 disk.nvme0n1.in‚ 92.38671875) disk./.usagePercent‚70.51366201025824 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 cpu‚25.326055376389817 disk.sda2.in‚0.03125 disk.rbd1.out‚ 64.5322265625( proc.memory.availableMB‚ 1942.1796875 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0ë«Zº”΂(:‰ õÙŠÓ™Òä³ gpu.0.gpu‚50.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.614' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.538 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿu$BÏ‚(:·  õÙŠÓÀÂ…´ disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0) proc.memory.availableMB‚ 1942.92578125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 322.23828125 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.785252888997395) disk./.usagePercent‚70.51366201025824 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚39206.236328125 proc.cpu.threads‚17% disk./.usageGB‚1260.8558921813965 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 64.5322265625 disk.drbd1285.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 network.recv‚786418 disk.dm-0.in‚ 92.38671875! disk.dm-0.out‚39206.236328125 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 network.sent‚ 1.2183814e+07 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0$ memory_percent‚92.09421475728354 proc.memory.rssMB‚ 2896.34375 disk.drbd2078.out‚0 disk.sda.out‚ 322.23828125 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1510.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 92.38671875 cpu‚25.325983913519092O¡ ªžÐ‚(:“ „ÚŠÓ̹³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.151' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.71700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íšsCÑ‚(:¸  „ÚŠÓñÂÛ³ disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0# disk.dm-0.out‚39206.55126953125 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 64.564453125 disk.drbd1550.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚39206.55126953125 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1510.in‚0 network.recv‚787336 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.785252888997395 network.sent‚ 1.2186263e+07 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0$ memory_percent‚92.09423065185547 cpu‚25.325914201864823 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 92.38671875 disk.drbd1510.out‚0 disk.sda.out‚ 322.49609375 proc.memory.rssMB‚ 2896.34375 disk.drbd1938.in‚0) disk./.usagePercent‚70.51366115690686% disk./.usageGB‚1260.8558769226074 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1942.921875 disk.sda2.out‚ 322.49609375 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.sda.in‚0.03125ƒY:V4Ò/ _wandbruntime‚12482  _runtime‚12482¦„¸ÓÇÓÁ & lr-AdamW/pg1‚6.246394803386043e-05 & lr-AdamW/pg2‚6.246394803386043e-05 trainer/global_step‚6849 $ _timestamp‚1.784851718706006e+09 _runtime‚12482.063225121 _step‚568ù?(ÌÔÆ $ _timestamp‚1.784851718706006e+09 _runtime‚12482.063225121 _step‚568 & lr-AdamW/pg1‚6.246394803386043e-05 & lr-AdamW/pg2‚6.246394803386043e-05 trainer/global_step‚6849¸¹‰M4Õ/ _wandbruntime‚12482  _runtime‚12482Ãòü(Ö‚P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7147255480320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7147255480320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7148600916480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18254116 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63597525 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39125925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20871455 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7148600916480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8742772340224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚539683790212096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8742721073152 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1979221566345728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚12482.064560802 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63607589 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548426930643456 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1978431603391278 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1971294945715728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7135287636296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39125566 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63607589 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 trainer/global_step‚6849 % _timestamp‚1.7848517187070327e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1972072939255808 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1979220194736128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59464732 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59464726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18254111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚539684158303232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1979220194736128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1979221566345728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1972072939255808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864æ›ñׇP [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7148600916480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8742772340224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚539683790212096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8742721073152 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1979221566345728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batcd[#ª}h_start/segment.small_pool.current‚824 _runtime‚12482.064560802 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63607589 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548426930643456 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1978431603391278 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1971294945715728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7135287636296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39125566 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63607589 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 trainer/global_step‚6849 % _timestamp‚1.7848517187070327e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1972072939255808 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1979220194736128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59464732 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59464726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18254111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚539684158303232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1979220194736128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1979221566345728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1972072939255808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7147255480320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7147255480320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7148600916480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18254116 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63597525 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39125925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20871455¹1ˆ©4Ø/ _wandbruntime‚12483  _runtime‚12483}¸­>'ÙŒN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123079700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59473099 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123089784 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63606607 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39131113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63616685 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7136308437936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59473093 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚539762744669184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7149623431680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20874362 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123089784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848517201820145e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1971573747489504 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1972351874719744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59473093 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7149623431680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1972351874719744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18256751 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1971573772917472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7148277988352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _step‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1978710055927440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548506344838144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚12483.539632249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚349 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005òòq'Ú‘N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7149623431680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20874362 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123089784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848517201820145e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1971573747489504 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1972351874719744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59473093 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7149623431680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1972351874719744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18256751 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1971573772917472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7148277988352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _step‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1978710055927440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548506344838144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚12483.539632249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚349 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123079700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59473099 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123089784 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63606607 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39131113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63616685 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7136308437936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59473093 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚539762744669184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824ºÅš›“4Û/ _wandbruntime‚12483  _runtime‚12483ÜsÖ Üƒ 1loss/FBt1/assignment_loss‚5.5821494346229e-05 /loss/FBt1/detection_loss‚0.7931568996213815 epoch‚24 _step‚571 0loss/FRt2/assignment_loss‚0.7446777877156381 1loss/SRqqt2/detection_loss‚0.6296005458364091 0loss/FBt2/detection_loss‚0.41034402708974854 %loss/total_loss‚8.173090774114733 trainer/global_step‚6849 % _timestamp‚1.7848517202198982e+09 _runtime‚12483.577155332 2loss/SRqqt2/assignment_loss‚0.4649391275130624 3loss/FBt2/assignment_loss‚0.0027130357625481352 /loss/FRt1/detection_loss‚0.5426620389633559 .loss/FRt2/detection_loss‚1.464082473061151 0loss/SRqqt1/detection_loss‚0.836946684014593 /loss/FRt1/assignment_loss‚1.652564172929036 2loss/SRqqt1/assignment_loss‚0.6313481601134632Á"ï݈ 2loss/SRqqt2/assignment_loss‚0.4649391275130624 3loss/FBt2/assignment_loss‚0.0027130357625481352 /loss/FRt1/detection_loss‚0.5426620389633559 .loss/FRt2/detection_loss‚1.464082473061151 0loss/SRqqt1/detection_loss‚0.836946684014593 /loss/FRt1/assignment_loss‚1.652564172929036 2loss/SRqqt1/assignment_loss‚0.6313481601134632 1loss/FBt1/assignment_loss‚5.5821494346229e-05 /loss/FBt1/detection_loss‚0.7931568996213815 epoch‚24 _step‚571 0loss/FRt2/assignment_loss‚0.7446777877156381 1loss/SRqqt2/detection_loss‚0.6296005458364091 0loss/FBt2/detection_loss‚0.41034402708974854 %loss/total_loss‚8.173090774114733 trainer/global_step‚6849 % _timestamp‚1.7848517202198982e+09 _runtime‚12483.577155332»ÔZMžÞ‚(:“ “ÚŠÓß‹ä³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚280.167' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.38899999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04€N>ß‚(:³  “ÚŠÓÊ÷„´ proc.memory.rssMB‚ 2896.34375 proc.cpu.threads‚17 disk.drbd1965.out‚0 network.recv‚788626 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 network.sent‚ 1.2191848e+07 disk.rbd1.in‚0& disk.nvme0n1.out‚39206.76220703125 disk.drbd1673.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚39206.76220703125 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 64.837890625 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.sda.out‚ 323.0078125 cpu‚25.32584191801096% disk./.usageGB‚1260.8558769226074 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.785252888997395) disk./.usagePercent‚70.51366115690686 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd2118.in‚0' proc.memory.availableMB‚ 1942.890625 disk.nvme0n1.in‚ 92.38671875 disk.drbd1584.out‚0 disk.sda2.out‚ 323.0078125 disk.dm-0.in‚ 92.38671875 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125" memory_percent‚92.094357808431 disk.drbd1938.in‚0H2žFŸà‚(:” ¢ÚŠÓ‘Ôʳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.662' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.55399999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏpYá‚(:´  ¢ÚŠÓ®îé³ disk.dm-0.in‚ 92.38671875 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1550.out‚0 disk.sda1.in‚0" memory_percent‚92.094357808431) disk./.usagePercent‚70.51366222359609 disk.drbd1817.in‚0' proc.memory.availableMB‚ 1942.890625 proc.memory.rssMB‚ 2896.34375% disk./.usageGB‚1260.8558959960938 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.dr¶=Õ,bd1552.out‚0 disk.drbd1673.out‚0" disk.dm-0.out‚39207.1630859375 disk.drbd1965.out‚0 disk.rbd1.out‚ 64.900390625 disk.drbd1285.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.785252888997395 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚39207.1630859375 disk.drbd1673.in‚0 proc.cpu.threads‚17 network.sent‚ 1.2249748e+07 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 323.31640625 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1584.in‚0 cpu‚25.325791956503394 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 network.recv‚790540 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.sda.out‚ 323.31640625 disk.drbd1289.in‚0 disk.drbd2118.out‚0 disk.drbd1828.in‚0±wµÔžâ‚(:“ ±ÚŠÓëƒá³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚281.785' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.92833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«“ƒ­=ã‚(:²  ±ÚŠÓͧ…´ disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0' proc.memory.availableMB‚ 1942.859375 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 323.82421875 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 network.recv‚792563 disk.rbd1.in‚0 disk.drbd1552.in‚0 cpu‚25.325700548889 disk.sda2.in‚0.03125 disk.drbd1285.out‚0 network.sent‚ 1.2254986e+07 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 92.38671875% disk.nvme0n1.out‚39207.3544921875 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51366222359609% disk./.usageGB‚1260.8558959960938 disk.dm-0.in‚ 92.38671875" disk.dm-0.out‚39207.3544921875 disk.drbd2111.out‚0 disk.drbd1817.out‚0# memory_percent‚92.0944849650065 proc.memory.rssMB‚ 2896.34375 disk.drbd1965.in‚0 disk.rbd1.out‚ 64.900390625 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 323.82421875 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.785252888997395ùÃ=žä‚(:“ ÀÚŠÓ‹–ȳ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.164' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.05466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜L^¢?å‚(:´  ÀÚŠÓøåî³ disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0* proc.memory.percent‚11.785252888997395 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.sda.out‚ 325.08984375 disk.dm-0.in‚ 92.38671875 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0# memory_percent‚92.0944849650065 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1673.out‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.51365795683925% disk./.usageGB‚1260.8558197021484 disk.drbd1550.out‚0' proc.memory.availableMB‚ 1942.859375 proc.cpu.threads‚17% disk.nvme0n1.out‚39207.9013671875 disk.sda1.out‚0 disk.drbd1584.out‚0 cpu‚25.3256507369778 disk.drbd1938.in‚0 disk.sda2.in‚0.03125 network.recv‚793587 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 325.08984375 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 network.sent‚ 1.2256011e+07 disk.drbd1931.out‚0" disk.dm-0.out‚39207.9013671875 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 65.0263671875 disk.drbd1510.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1817.in‚0à)ãNžæ‚(:“ ÏÚŠÓî§Ò³ gpu.0.gpu‚29.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚107.914' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.971333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EC„Å?ç‚(:´  ÏÚŠÓÞõ³ disk.drbd1550.out‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 92.38671875" disk.dm-0.out‚39208.0615234375 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0# memory_percent‚92.0944849650065 disk.dm-0.in‚ 92.38671875 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd2041.out‚0% disk./.usageGB‚1260.8558959960938 disk.rbd1.out‚ 65.1279296875 disk.drbd1356.out‚0 disk.sda.out‚ 325.6171875) disk./.usagePercent‚70.51366222359609 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 network.sent‚ 1.2261111e+07 disk.drbd1965.out‚0 disk.drbd1510.out‚0 network.recv‚794724% disk.nvme0n1.out‚39208.0615234375 disk.drbd2118.in‚0 disk.sda2.out‚ 325.6171875 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 cpu‚25.325600103309743 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0' proc.memory.availableMB‚ 1942.859375* proc.memory.percent‚11.785252888997395Ì .4è/ _wandbruntime‚12568  _runtime‚12568:Ë@ÆéÀ & lr-AdamW/pg2‚6.179469270244255e-05 trainer/global_step‚6899 % _timestamp‚1.7848518052672963e+09 _runtime‚ 12568.6245434 _step‚572 & lr-AdamW/pg1‚6.179469270244255e-05ÖËz¯ËêÅ % _timestamp‚1.7848518052672963e+09 _runtime‚ 12568.6245434 _step‚572 & lr-AdamW/pg1‚6.179469270244255e-05 & lr-AdamW/pg2‚6.179469270244255e-05 trainer/global_step‚6899¼89«4ë/ _wandbruntime‚12568  _runtime‚12568Žûu(ìP A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1986018403780608 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8805319080960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64063541 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39404050 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 _step‚573 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123946623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1992422795962886 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1985235059832496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64053477 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39404396 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7198445036544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1993216848817152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21017939 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6899 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1993218220426752 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64063541 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1986018429954048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7187736130390 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848518052684402e+09 _runtime‚12568.62615484 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1986018429954048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552418840657408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21018280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123936553 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18386116 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543613476600832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1985235034404528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7199790472704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8805364056576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1993216848817152 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚552418427590656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64063541 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39404050 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 _step‚573 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123946623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1992422795962886 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1985235059832496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64053477 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39404396 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7198445036544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1993216848817152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21017939 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6899 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1993218220426752 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64063541 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1986018429954048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7187736130390 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848518052684402e+09 _runtime‚12568.62615484 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1986018429954048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552418840657408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21018280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123936553 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18386116 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543613476600832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1985235034404528½h:›>î‚(:³  ÞÚŠÓ½Ñø³ disk.drbd2111.out‚0 disk.drbd1584.out‚0# disk./.usageGB‚1260.85591506958 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1943.109375* proc.memory.percent‚11.785252888997395( disk./.usagePercent‚70.5136632902853 disk.dm-0.in‚ 92.38671875 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 65.1904296875 disk.nvme0n1.in‚ 92.38671875& disk.nvme0n1.out‚39208.66552734375 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚39208.66552734375 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.sda.in‚0.03125 disk.rbd1.in‚0 cpu‚25.325490185348 network.recv‚794856 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.sda2.out‚ 325.85546875 disk.drbd1285.out‚0 disk.drbd1817.out‚0$ memory_percent‚92.09346771240234 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 325.85546875 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 proc.memory.rssMB‚ 2896.34375 network.sent‚ 1.2261923e+07 disk.drbd1965.in‚0uœ®žï‚(:“ ÞÚŠÓ§«Ô³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.468' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.82266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/_§ù4ð/ _wandbruntime‚12570  _runtime‚125702œê¬ ñŒN ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21020857 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64072637 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64072637 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21021195 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1992702618538414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59891443 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1993498152232448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18388756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543691694875648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1985513836178304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7187412320744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1993498152232448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552498694111744 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1985513861606272 % _timestamp‚1.7848518067441814e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚338 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123954002 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543692062966784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1992701248499048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39409951 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64062559 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1986297313071104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7199467544576 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _step‚574 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7199467544576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123964086 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123964086 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39409608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1986297339244544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7200812987904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8806586176512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123954002 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543692062966784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1992701248499048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39409951 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64062559 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1986297313071104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7199467544576 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _step‚574 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7199467544576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123964086 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123964086 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39409608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1986297339244544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7200812987904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8806586176512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1992702618538414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59891443 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1993498152232448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18388756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543691694875648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1985513836178304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7187412320744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1993498152232448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552498694111744 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1985513861606272¾ÛM•|4ó/ _wandbruntime‚12570  _runtime‚12570g[×÷ ô† %loss/total_loss‚7.862627859457814 trainer/global_step‚6899 % _timestamp‚1.7848518067802134e+09 0loss/FRt1/assignment_loss‚1.6425693768766982 0loss/FRt2/assignment_loss‚0.7084268956956729 /loss/FRt2/detection_loss‚1.3870211502962349 1loss/SRqqt1/detection_loss‚0.8607542626646504 1loss/SRqqt2/detection_loss‚0.6701662573190531 /loss/FBt1/detection_loss‚0.5343003805933727 2loss/SRqqt2/assignment_loss‚0.4870265891290478 2loss/FBt2/assignment_loss‚0.009236031068260112 /loss/FBt2/detection_loss‚0.5231655858467035 _runtime‚12570.137452634 epoch‚24 3loss/FBt1/assignment_loss‚6.963139870386215e-05 /loss/FRt1/detection_loss‚0.5308750133464285 _step‚575 2loss/SRqqt1/assignment_loss‚0.5090166852229875½5)¾õ‹ epoch‚24 3loss/FBt1/assignment_loss‚6.963139870386215e-05 /loss/FRt1/detection_loss‚0.5308750133464285 _step‚575 2loss/SRqqt1/assignment_loss‚0.5090166852229875 %loss/total_loss‚7.862627859457814 trainer/global_step‚6899 % _timestamp‚1.7848518067802134e+09 0loss/FRt1/assignment_loss‚1.6425693768766982 0loss/FRt2/assignment_loss‚0.7084268956956729 /loss/FRt2/detection_loss‚1.3870211502962349 1loss/SRqqt1/detection_loss‚0.8607542626646504 1loss/SRqqt2/detection_loss‚0.6701662573190531 /loss/FBt1/detection_loss‚0.5343003805933727 2loss/SRqqt2/assignment_loss‚0.4870265891290478 2loss/FBt2/assignment_loss‚0.009236031068260112 /loss/FBt2/detection_loss‚0.5231655858467035 _runtime‚12570.137452634¿›³óîžö‚(:“ íÚŠÓ‘Øг gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.508' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.83599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁŒÂLB÷‚(:·  íÚŠÓõï³ disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 proc.memory.rssMB‚ 2896.34375 disk.dm-0.in‚ 92.38671875 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0# disk.dm-0.out‚39208.90380859375 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 cpu‚25.325460157205924 proc.cpu.threads‚17 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1226.in‚0* proc.memory.percent‚11.785252888997395 disk.drbd1226.out‚0 disk.drbd1550.out‚0( proc.memory.availableMB‚ 1942.7421875 disk.drbd1817.out‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 326.2421875# memory_percent‚92.0949618021647 disk.drbd1817.in‚0 disk.rbd1.out‚ 65.1904296875 disk.drbd1285.in‚0 disk.drbd1510.in‚0 network.sent‚ 1.2324491e+07 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 network.recv‚798494% disk./.usageGB‚1260.8548545837402 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚39208.90380859375 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51360398236552 disk.drbd1673.out‚0 disk.sda2.out‚ 326.2421875 disk.drbd1289.out‚0 disk.drbd1965.out‚0jBø‚(:·  üÚŠÓ“³ï³ disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 65.22265625 disk.drbd1931.out‚0 disk.drbd1938.out‚0# disk.dm-0.out‚39209.80126953125 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 cpu‚25.32541016225855 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0* proc.memory.percent‚11.785252888997395 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 326.55859375 disk.drbd1285.in‚0$ disk./.usageGB‚1260.854835510254 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0& disk.nvme0n1.out‚39209.80126953125 network.sent‚ 1.2325474e+07) disk./.usagePercent‚70.51360291567632 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 92.38671875) proc.memory.availableMB‚ 1942.49609375 proc.memory.rssMB‚ 2896.34375 disk.drbd1550.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0$ memory_percent‚92.09596316019693 disk.drbd1817.out‚0 network.recv‚799311 disk.sda.out‚ 326.55859375 disk.sda2.in‚0.03125 disk.dm-0.in‚ 92.38671875¨&ú†žù‚(:“ üÚŠÓü¬Í³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚121.108' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.36933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hèqëžú‚(:“ ‹ÛŠÓ—™¯³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚256.244' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.41466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶÷E®Bû‚(:·  ‹ÛŠÓãËÓ³ disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 92.38671875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 cpu‚25.325321640687136 disk.dm-0.in‚ 92.38671875 disk.drbd2111.out‚0& disk.nvme0n1.out‚39210.14501953125 disk.drbd1584.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51360291567632 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 327.03515625 proc.memory.rssMB‚ 2896.34375 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0# disk.dm-0.out‚39210.14501953125 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0$ disk./.usageGB‚1260.854835510254 disk.rbd1.in‚0) proc.memory.availableMB‚ 1942.49609375 proc.cpu.threads‚17 disk.sda2.in‚0.03125* proc.memory.percent‚11.785252888997395 disk.drbd2041.out‚0 disk.drbd1673.out‚0$ memory_percent‚92.09596316019693 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 65.41015625 disk.sda2.out‚ 327.03515625 disk.drbd1289.out‚0 network.recv‚800366 disk.drbd2078.out‚0 network.sent‚ 1.232793e+07 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚09 iñžü‚(:“ šÛŠÓæ˨³ gpu.0.gpu‚96.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.162' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.72066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ãÔî?ý‚(:´  šÛŠÓùödz disk.drbd2041.in‚0) disk./.usagePercent‚70.51360291567632 disk.rbd1.out‚ 65.47265625) proc.memory.availableMB‚ 1942.49609375 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 proc.memory.rssMB‚ 2896.34375 cpu‚25.325272287057434 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125! disk.dm-0.out‚39210.599609375 disk.drbd1828.out‚0 disk.sda2.out‚ 327.42578125 disk.dm-0.in‚ 92.70703125 disk.drbd1285.in‚0$ disk.nvme0n1.out‚39210.599609375 network.sent‚ 1.2333528e+07$ disk./.usageGB‚1260.854835510254 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.sda1.in‚0 network.recv‚800738 disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 327.42578125 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.09596316019693 disk.drbd1285.out‚0* proc.memory.percent‚11.785252888997395 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.nvme0n1.in‚ 92.70703125h°DÞþ‚(:“ ©ÛŠÓçÔí³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚109.177' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.39233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vëì¬8ÿ‚(:­  ©ÛŠÓ¸¨´ disk.drbd1285.in‚0 disk.drbd1289.in‚0! disk.dm-0.out‚39210.794921875 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1226.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.785252888997395 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 65.47265625 cpu‚25.32522029666861 disk.drbd1817.out‚0 disk.dm-0.in‚ 92.70703125 network.recv‚800804 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2078.out‚0$ memory_percent‚92.09596316019693 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 327.78125 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 network.sent‚ 1.2333594e+07) disk./.usagePercent‚70.51360291567632 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd2041.in‚0$ disk.nvme0n1.out‚39210.794921875) proc.memory.availableMB‚ 1942.49609375$ disk./.usageGB‚1260.854835510254 disk.sda.out‚ 327.78125óÖ4€/ _wandbruntime‚12655  _runtime‚12655(`È _step‚576 & lr-AdamW/pg1‚6.112501396827872e-05 & lr-AdamW/pg2‚6.112501396827872e-05 trainer/global_step‚6949 % _timestamp‚1.7848518920309448e+09 _runtime‚12655.388182525Ã%ç Í‚Ç & lr-AdamW/pg1‚6.112501396827872e-05 & lr-AdamW/pg2‚6.112501396827872e-05 trainer/global_step‚6949 % _timestamp‚1.7848518920309448e+09 _runtime‚12655.388182525 _step‚576Àä4ƒ/ _wandbruntime‚12655  _runtime‚12655EÈ ‘V „‚P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556410386372096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7249634592768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7250980028928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1999963868305408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60301432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556410809924608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚547542794898432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7237495401912 _runtime‚12655.392970468 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1999963894478848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39682500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 MGDeviceStatsMonitor.on_train_†(^d²batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7238840013198 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64519493 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7250980028928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1999175123093328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚6949 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18518111 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7249634592768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8868015026176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60301426 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2007214874507776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848518920320187e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8867959564800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39682139 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18518116 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547542426807296 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2006413988534494 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1999175148521296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1999963894478848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60301426 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2007213502898176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124820925 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2007213502898176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21164028 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2007214874507776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653‚<‚D (…‡P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2007213502898176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21164028 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2007214874507776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556410386372096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7249634592768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7250980028928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1999963868305408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60301432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556410809924608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚547542794898432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7237495401912 _runtime‚12655.392970468 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1999963894478848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39682500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7238840013198 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64519493 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7250980028928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1999175123093328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚6949 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18518111 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7249634592768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8868015026176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60301426 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2007214874507776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848518920320187e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8867959564800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39682139 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18518116 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547542426807296 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2006413988534494 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1999175148521296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1999963894478848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60301426 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2007213502898176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124820925 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181ÁìÔ²4†/ _wandbruntime‚12656  _runtime‚12656~“Y…'‡ŒN EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2000242803769344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18520751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252002544128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1999453950295072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124838388 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64518511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39687560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2006692441070656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _step‚578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8869267102720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21166809 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2006693811110022 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6949 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚547621013173248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181Ý7qÆ؈‘N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60309799 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848518935091906e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124838388 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64518511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39687560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2006692441070656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _step‚578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8869267102720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21166809 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2006693811110022 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6949 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚547621013173248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2000242803769344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18520751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252002544128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1999453950295072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F—‚(:³  ƒÜŠÓòºÒ³ disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0" disk.dm-0.out‚39212.6435546875 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 329.7421875$ disk./.usageGB‚1260.854824066162 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 329.7421875 disk.rbd1.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2896.34375 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 92.70703125% disk.nvme0n1.out‚39212.6435546875 disk.drbd1584.out‚0) proc.memory.availableMB‚ 1942.80859375 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 network.sent‚ 1.240611e+07 disk.dm-0.in‚ 92.70703125 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 proc.cpu.threads‚17 network.recv‚807221 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.785252888997395 disk.drbd2111.out‚0 disk.rbd1.out‚ 66.0986328125( disk./.usagePercent‚70.5136022756628 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 cpu‚25.32500537991987$ memory_percent‚92.09469159444174 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0LXa4˜/ _wandbruntime‚12742  _runtime‚12742 ²DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21308955 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2013909332830208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2013115237210096 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2021211528588800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551472113196032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8930475604480 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64965381 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚350 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2020405181106102 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7302169585152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60719776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18650111 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60719782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 trainer/global_step‚6999 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7300824148992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60719776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21309300 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚345 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60719782 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2020403811066848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64975445 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2021210156979200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7289943896006 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2013909359003648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64975445 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2013115211782128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125695227 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768Ït&Y° †P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7288599284720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8930522677248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125685157 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39959066 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21309300 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚345 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60719782 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2020403811066848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64975445 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2021210156979200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7289943896006 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allo8$t\cated‚2013909359003648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64975445 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2013115211782128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125695227 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚551471745104896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7300824148992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21308955 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2013909332830208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2013115237210096 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2021211528588800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551472113196032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8930475604480 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64965381 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚350 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2020405181106102 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7302169585152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60719776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18650111 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60719782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 trainer/global_step‚6999 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7300824148992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60719776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7289620086360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7301846657024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39964963 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125712690 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125712690 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2020685003681630 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7303192100352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60728149 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560482059495936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8931775090688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7303192100352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64974463 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39964615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60728149 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6999 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64974463 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2014188242120704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21311864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2021491460394496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚348 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _step‚582ت]c' N SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6999 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64974463 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2014188242120704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21311864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2021491460394496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚348 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _step‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18652751 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125702606 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64984541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 $ _timestamp‚1.784851980185495e+09 _runtime‚12743.543095472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60728143 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60728143 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7290964697758 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7289620086360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7301846657024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39964963 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125712690 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125712690 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2020685003681630 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7303192100352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60728149 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560482059495936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8931775090688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7303192100352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64974463 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39964615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60728149 F&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25519150495529175 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5766178965568542 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7059460282325745 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12839052081108093 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43890246748924255 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8803033828735352 particle/accuracy_1_of_3‚1 $ _timestamp‚1.784851999718384e+09 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2712969183921814 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5012303590774536 +jet/accuracy_3_of_4‚0.10971427708864212 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692209124565125 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7173824310302734 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1`­F‘3 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7018375396728516 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6145809292793274 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5307692289352417 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3231138586997986 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27324455976486206 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7975159883499146 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1302635371685028 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7068080306053162 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35131949186325073 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7135060429573059 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7324609756469727 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.529359757900238 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.33165809512138367 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2738521993160248 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974471926689148 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2618458569049835 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27631011605262756 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9412111639976501 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8250231742858887 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1588733196258545 *jet/accuracy_2_of_4‚0.8379398584365845 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333730697632 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6811472773551941 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3315221965312958 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7585316896438599 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21791349351406097 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8026930689811707 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7474949955940247 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚6999 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_3‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8225571513175964 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925484657287598 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021732157096266747 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4455938935279846 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7571021318435669 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04386860504746437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.72569739818573 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9151220321655273 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3315221965312958 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7301986813545227 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26854217052459717 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8122857213020325 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7376445531845093 *jet/accuracy_3_of_3‚0.0734100416302681 particle/accuracy_2_of_2‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8723620176315308 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 *jet/accuracy_1_of_4‚0.9743425250053406 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6455857157707214 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9898831248283386 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8172167539596558 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38298413157463074 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4369412362575531 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.36143046617507935 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4112391173839569 _step‚584 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8213514685630798 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6107468605041504 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024161534383893013 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8542190194129944 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15156103670597076 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 epoch‚24 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6122385263442993 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40692588686943054 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8243918418884277 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7457637190818787 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020029151812195778 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 +jet/accuracy_1_of_1‚0.37636345624923706 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7608052492141724 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5113115310668945 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 /particle/accuracy_3_of_4‚0.9964285492897034 +particle/specificity‚0.9126740097999573 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99635249376297 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 *jet/accuracy_2_of_3‚0.6053852438926697 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22137339413166046 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8182544708251953 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 particle/accuracy_1_of_1‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3435540795326233 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08004502207040787 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6651691794395447 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026568790897727013 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3307490646839142 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 _runtime‚12763.07643989 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3836126923561096 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878942370414734 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6056532263755798 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6866162419319153 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.052485425025224686 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8803033828735352 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.669277012348175 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2520635426044464 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7429104447364807 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16369876265525818 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7692494988441467 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.988954677019352 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28378885984420776 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9928030371665955 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6866162419319153 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16958744823932648 +particle/sensitivity‚0.6402220129966736 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6904857754707336 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6782588958740234 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7131518721580505 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 /particle/accuracy_4_of_4‚0.9730225801467896 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7701687812805176 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3427933156490326 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 'particle/f_score‚0.6533126831054688 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7674464583396912 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28468790650367737 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8084298968315125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9390355348587036 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14243002235889435 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7653846144676208 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4078585207462311 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.30018314719200134 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6850057244300842 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34517449140548706 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7743996977806091 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06404527276754379 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28378620743751526 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675ÌdA¤2;¨¬v 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9928030371665955 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6866162419319153 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16958744823932648 +particle/sensitivity‚0.6402220129966736 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6904857754707336 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6782588958740234 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7131518721580505 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 /particle/accuracy_4_of_4‚0.9730225801467896 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7701687812805176 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3427933156490326 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 'particle/f_score‚0.6533126831054688 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7674464583396912 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28468790650367737 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8084298968315125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9390355348587036 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14243002235889435 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7653846144676208 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4078585207462311 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.30018314719200134 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6850057244300842 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34517449140548706 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7743996977806091 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06404527276754379 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28378620743751526 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7011470794677734 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8190708160400391 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25519150495529175 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5766178965568542 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7059460282325745 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12839052081108093 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43890246748924255 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8803033828735352 particle/accuracy_1_of_3‚1 $ _timestamp‚1.784851999718384e+09 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2712969183921814 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5012303590774536 +jet/accuracy_3_of_4‚0.10971427708864212 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692209124565125 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7173824310302734 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7018375396728516 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6145809292793274 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5307692289352417 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3231138586997986 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27324455976486206 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7975159883499146 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1302635371685028 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7068080306053162 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35131949186325073 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7135060429573059 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7324609756469727 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.529359757900238 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.33165809512138367 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2738521993160248 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974471926689148 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2618458569049835 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27631011605262756 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9412111639976501 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8250231742858887 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1588733196258545 *jet/accuracy_2_of_4‚0.8379398584365845 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333730697632 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6811472773551941 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3315221965312958 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7585316896438599 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21791349351406097 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8026930689811707 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7474949955940247 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚6999 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_3‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8225571513175964 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925484657287598 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021732157096266747 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4455938935279846 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7571021318435669 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04386860504746437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.72569739818573 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9151220321655273 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3315221965312958 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7301986813545227 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26854217052459717 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8122857213020325 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7376445531845093 *jet/accuracy_3_of_3‚0.0734100416302681 particle/accuracy_2_of_2‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8723620176315308 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 *jet/accuracy_1_of_4‚0.9743425250053406 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6455857157707214 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9898831248283386 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8172167539596558 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38298413157463074 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4369412362575531 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.36143046617507935 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4112391173839569 _step‚584 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8213514685630798 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6107468605041504 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024161534383893013 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8542190194129944 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15156103670597076 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 epoch‚24 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6122385263442993 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40692588686943054 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8243918418884277 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7457637190818787 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020029151812195778 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 +jet/accuracy_1_of_1‚0.37636345624923706 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7608052492141724 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5113115310668945 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 /particle/accuracy_3_of_4‚0.9964285492897034 +particle/specificity‚0.9126740097999573 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99635249376297 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 *jet/accuracy_2_of_3‚0.6053852438926697 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22137339413166046 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8182544708251953 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 particle/accuracy_1_of_1‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3435540795326233 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08004502207040787 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6651691794395447 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026568790897727013 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3307490646839142 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 _runtime‚12763.07643989 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3836126923561096 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878942370414734 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6056532263755798 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6866162419319153 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.052485425025224686 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8803033828735352 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.669277012348175 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2520635426044464 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7429104447364807 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16369876265525818 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7692494988441467 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.988954677019352 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28378885984420776È3²9 ©‚(:’ ¡ÜŠÓˆðì³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚58.0 gpu.0.powerWatts‚106.102' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.367333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,¯bIEª‚(:º  ¡ÜŠÓ¥œˆ´ disk.drbd1673.out‚0 network.recv‚808433) disk./.usagePercent‚70.57378253385006 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 330.42578125 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 network.sent‚ 1.2406661e+07 disk.drbd2111.in‚0 disk.drbd2118.out‚0 cpu‚25.328558708430215 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 3872.80078125 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd2041.out‚0& disk.nvme0n1.out‚39213.10693359375* proc.memory.percent‚15.758466720581055$ memory_percent‚99.97843106587729$ disk./.usageGB‚1261.930908203125 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 330.42578125& proc.memory.availableMB‚ 5.30078125 disk.rbd1.out‚ 66.1611328125 disk.nvme0n1.in‚ 92.70703125 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0# disk.dm-0.out‚39213.10693359375 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0#z¬2ž«‚(:“ °ÜŠÓ£‚¿³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚293.018' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.67266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ræ:D¬‚(:¹  °ÜŠÓŽàá³ disk.sda2.out‚ 330.8828125) disk./.usagePercent‚70.51381369346308 disk.nvme0n1.in‚ 92.70703125$ memory_percent‚88.14541498819986 network.recv‚808499 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0) proc.memory.percent‚11.81344985961914 network.sent‚ 1.2406727e+07 disk.sda.in‚0.03125 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚40183.50830078125 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚40183.50830078125 disk.drbd1226.out‚0 disk.sda.out‚ 330.8828125 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2111.out‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 2913.3828125% disk./.usageGB‚1260.8586044311523 disk.drbd1226.in‚0 disk.drbd1817.out‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 66.4033203125 cpu‚25.327511963263692 disk.dm-0.in‚ 92.70703125 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0$çÀ H­‚(:’ ¿ÜŠÓؽѳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚297.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.17333333333332 gpu.0.smClock‚1380 gpu.0.memorŸÉ_UyClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aä´G®‚(:¼  ¿ÜŠÓžßü³ disk.sda2.in‚0.03125 disk.drbd1289.out‚0% disk./.usageGB‚1260.8587036132812 disk.drbd1584.in‚0 disk.sda.in‚0.03125 network.recv‚813143 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 66.5048828125 disk.sda2.out‚ 331.19140625 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚40183.85595703125 disk.drbd1510.in‚0 cpu‚25.327459429947304 disk.sda1.out‚0$ memory_percent‚88.14441363016763 disk.drbd1817.in‚0 disk.drbd1965.out‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1817.out‚0 disk.sda.out‚ 331.19140625 disk.dm-0.in‚ 92.70703125 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 92.70703125 proc.cpu.threads‚17 disk.drbd1938.out‚0) proc.memory.percent‚11.81344985961914& disk.nvme0n1.out‚40183.85595703125 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0) disk./.usagePercent‚70.51381924024695 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 2913.62890625 network.sent‚ 1.2494756e+07 disk.drbd2041.in‚0÷&á2¯‚(:’ ÎÜŠÓúéâ³ gpu.0.gpu‚57.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚107.03' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.67666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n¹îòA°‚(:¶  ÎÜŠÓ¯ü†´ disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.dm-0.out‚41153.79296875 disk.drbd1965.in‚0% disk./.usageGB‚1260.8549041748047 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51360675575746 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 92.70703125 network.recv‚813251 disk.rbd1.out‚ 66.5673828125 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 331.60546875 disk.drbd2041.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0# disk.nvme0n1.out‚41153.79296875 network.sent‚ 1.2494864e+07 disk.sda1.in‚0$ memory_percent‚88.14555803934732" proc.memory.rssMB‚ 2903.2734375 disk.drbd1550.out‚0 disk.drbd1289.in‚0 cpu‚25.327349169438204 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 2913.34765625 disk.drbd1285.in‚0 disk.drbd1673.out‚0) proc.memory.percent‚11.81344985961914 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 92.70703125 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 331.60546875 disk.sda2.in‚0.03125€NÏž±‚(:“ ÝÜŠÓÜòϳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.751' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.58366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E›u–;²‚(:°  ÝÜŠÓˆ×ñ³ disk.drbd1550.in‚0 cpu‚25.327279073483695 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 332.1953125 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125" proc.memory.rssMB‚ 2903.2734375 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.dm-0.out‚41153.95703125& proc.memory.availableMB‚ 2913.59375 disk.sda.out‚ 332.1953125 disk.drbd1938.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51360568906826 disk.drbd1552.out‚0 disk.drbd1285.out‚0# disk.nvme0n1.out‚41153.95703125% disk./.usageGB‚1260.8548851013184 disk.drbd1931.in‚0# memory_percent‚88.1445566813151 network.recv‚816016 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 66.5673828125 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1931.out‚0 disk.drbd1938.out‚0 network.sent‚ 1.2501214e+07 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.nvme0n1.in‚ 92.70703125 disk.sda1.out‚0 disk.drbd1584.out‚0) proc.memory.percent‚11.81344985961914XšªŽ?³‚(:´  ìÜŠÓ‰Ò³$ disk./.usageGB‚1260.854866027832 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 cpu‚25.327228884372087 proc.cpu.threads‚17) disk./.usagePercent‚70.51360462237906 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125) proc.memory.percent‚11.81344985961914 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚41154.64013671875 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 332.4296875 disk.drbd2078.out‚0 network.recv‚816148 disk.drbd1356.in‚0 disk.drbd2041.out‚0# memory_percent‚88.1445566813151 network.sent‚ 1.2502022e+07 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1356.out‚0& proc.memory.availableMB‚ 2913.59375 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0# disk.dm-0.out‚41154.64013671875 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 66.599609375 disk.sda.out‚ 332.4296875 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0åÙœ©ž´‚(:“ ìÜŠÓþ£°³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.257' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.75233333333333 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“^4µ/ _wandbruntime‚12853  _runtime‚12853´]ZÍȶ _step‚585 & lr-AdamW/pg1‚5.978473745386152e-05 & lr-AdamW/pg2‚5.978473745386152e-05 trainer/global_step‚7049 % _timestamp‚1.7848520901219633e+09 _runtime‚12853.479191462Ã:80Í·Ç & lr-AdamW/pg1‚5.978473745386152e-05 & lr-AdamW/pg2‚5.978473745386152e-05 trainer/global_step‚7049 % _timestamp‚1.7848520901219633e+09 _runtime‚12853.479191462 _step‚585Ɇ!; 4¸/ _wandbruntime‚12853  _runtime‚128536w¸ (¹‚P _runtime‚12853.480458415 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2036960983668192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2037773106224128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚341 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126696203 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2030415500253184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848520901230102e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61227564 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65478709 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9000437649920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7357632144384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40274539 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555450431006720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2037774477833728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7357632144384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61227564 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61227558 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7358977580544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40274193 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21485190 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7345312006842 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65468645 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2030415474079744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7346656618128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18789008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2030415500253184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65478709 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65468645 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126706273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564450500565504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2037773106224128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555450062915584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7358977580544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61227558 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2036959613628938 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2037774477833728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2029614301622096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2029614327050064 ¸ (º‡P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21485531 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2030415474079744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7346656618128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18789008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2030415500253184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65478709 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65468645 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126706273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564450500565504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2037773106224128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555450062915584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7358977580544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61227558 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2036959613628938 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2037774477833728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2029614301622096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2029614327050064 _runtime‚12853.480458415 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2036960983668192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2037773106224128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚341 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126696203 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2030415500253184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848520901230102e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61227564 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65478709 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9000437649920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7357632144384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40274539 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555450431006720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2037774477833728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7357632144384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61227564 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61227558 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7358977580544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40274193 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21485190 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7345312006842 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65468645 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2038053038022656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2029893128823840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65487805 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚12854.954799359 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2038054409639424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126723736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126713652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40280060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18791648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18791643 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848520915971541e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7347677419880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2030694409543680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2037240806243720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7358654652416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9001672681984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7360000095744 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2030694383370240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocatª™JÒøed‚ 126723736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61235931 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40279709 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65477727 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61235931 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2038054409639424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2030694383370240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚587 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2029893103395872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21488412 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚555528649281536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65487805 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21488412 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚555528649281536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65487805 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2038053038022656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2029893128823840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65487805 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚12854.954799359 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2038054409639424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126723736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126713652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40280060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18791648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18791643 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848520915971541e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7347677419880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2030694409543680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2037240806243720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7358654652416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9001672681984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7360000095744 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2030694383370240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126723736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61235931 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40279709 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65477727 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61235931 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2038054409639424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2030694383370240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚587 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2029893103395872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005ËÔ­†4¾/ _wandbruntime‚12854  _runtime‚12854”ëß ¿… 1loss/FBt2/assignment_loss‚0.01628782287610935 1loss/SRqqt1/detection_loss‚0.8840562983189569 /loss/FBt2/detection_loss‚0.4774334062658838 % _timestamp‚1.7848520916354167e+09 _runtime‚12854.992622775 2loss/SRqqt2/assignment_loss‚0.5316810541381426 /loss/FRt2/detection_loss‚1.4308580408467868 1loss/SRqqt2/detection_loss‚0.6703049342635573 /loss/FBt1/detection_loss‚0.6391445697864165 %loss/total_loss‚8.006721115976553 epoch‚25 trainer/global_step‚7049 _step‚588 0loss/FRt2/assignment_loss‚0.7205385395863277 /loss/FRt1/detection_loss‚0.6005642227022842 /loss/FRt1/assignment_loss‚1.569773330733731 3loss/SRqqt1/assignment_loss‚0.46557543678237345 3loss/FBt1/assignment_loss‚0.0005034596759829457H=`œÀŠ 0loss/FRt2/assignment_loss‚0.7205385395863277 /loss/FRt1/detection_loss‚0.6005642227022842 /loss/FRt1/assignment_loss‚1.569773330733731 3loss/SRqqt1/assignment_loss‚0.46557543678237345 3loss/FBt1/assignment_loss‚0.0005034596759829457 1loss/FBt2/assignment_loss‚0.01628782287610935 1loss/SRqqt1/detection_loss‚0.8840562983189569 /loss/FBt2/detection_loss‚0.4774334062658838 % _timestamp‚1.7848520916354167e+09 _runtime‚12854.992622775 2loss/SRqqt2/assignment_loss‚0.5316810541381426 /loss/FRt2/detection_loss‚1.4308580408467868 1loss/SRqqt2/detection_loss‚0.6703049342635573 /loss/FBt1/detection_loss‚0.6391445697864165 %loss/total_loss‚8.006721115976553 epoch‚25 trainer/global_step‚7049 _step‚588ÌhŒ ê@Á‚(:µ  ûÜŠÓÒ³ó³ network.recv‚817496 disk.drbd1285.in‚0 disk.drbd1226.in‚0 network.sent‚ 1.250708e+07 disk.dm-0.in‚ 92.70703125 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd2041.out‚0# disk.dm-0.out‚41154.81982421875 disk.drbd1673.in‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚41154.81982421875 disk.drbd1510.in‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51360696909529 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 cpu‚25.327137934659532 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1550.in‚0 disk.drbd2118.in‚0$ memory_percent‚88.14449310302734$ disk./.usageGB‚1260.854907989502 disk.sda.out‚ 332.8046875 disk.rbd1.out‚ 66.794921875 disk.drbd1584.out‚0 disk.sda2.out‚ 332.8046875' proc.memory.availableMB‚ 2913.609375 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0) proc.memory.percent‚11.813449859619140VJñžÂ‚(:“ ûÜŠÓ´ÑÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.835' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.27833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YBí“Â(:ˆ ŠÝŠÓËÓ³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚291.618' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oÚ¾=Ä‚(:²  ŠÝŠÓ»òú³ disk.drbd1550.out‚0 cpu‚25.327067399656464$ disk./.usageGB‚1260.854907989502 disk.sda2.out‚ 333.1953125 disk.dm-0.in‚ 92.70703125 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.rbd1.out‚ 66.857421875 disk.drbd1289.out‚0 disk.drbd1356.in‚0" proc.memory.rssMB‚ 2903.2734375 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0% disk.nvme0n1.out‚41155.4111328125 disk.drbd2118.in‚0 disk.drbd2118.out‚0$ memory_percent‚88.14570109049478 network.sent‚ 1.2507188e+07 disk.drbd1552.out‚0 network.recv‚817604 disk.drbd1552.in‚0 disk.drbd1673.in‚0) proc.memory.percent‚11.81344985961914 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 333.1953125 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1673.out‚0" disk.dm-0.out‚41155.4111328125 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0% proc.memory.availableMB‚ 2913.3125) disk./.usagePercent‚70.51360696909529 disk.sda2.in‚0.03125 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0f2’Å‚(:‡ ™ÝŠÓ¶õÖ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚301.38' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.46 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þi%AÆ‚(:¶  ™ÝŠÓâÃ÷³% disk.nvme0n1.out‚41155.5634765625 disk.drbd1226.out‚0 disk.dm-0.in‚ 92.70703125( disk./.usagePercent‚70.5136059024061 disk.drbd2041.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0) proc.memory.percent‚11.81344985961914 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 333.8359375 network.sent‚ 1.2507254e+07 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda.out‚ 333.8359375 disk.drbd1289.out‚0 network.recv‚817670 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0$ memory_percent‚88.14578056335449 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 66.857421875 disk.drbd1552.out‚0 disk.drbd1584.in‚0" disk.dm-0.out‚41155.5634765625 cpu‚25.327016719127617 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1673.in‚0) proc.memory.availableMB‚ 2913.29296875" proc.memory.rssMB‚ 2903.2734375 disk.drbd2078.in‚0% disk./.usageGB‚1260.8548889160156 disk.drbd1510.in‚0 disk.drbd1673.out‚0§MáNžÇ‚(:“ ¨ÝŠÓ£ûÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.613' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.20433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ a‚?È‚(:´  ¨ÝŠÓîâ³( disk./.usagePercent‚70.5136080357845 disk.drbd1965.out‚0) proc.memory.percent‚11.81344985961914 cpu‚25.326928082229433% disk./.usageGB‚1260.8549270629883 disk.dm-0.in‚ 92.70703125 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 2913.2890625 disk.drbd1550.in‚0$ memory_percent‚88.14579645792642" proc.memory.rssMB‚ 2903.2734375 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1550.out‚0 network.sent‚ 1.2507632e+07 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.drbd2078.in‚0# disk.nvme0n1.out‚41156.23046875 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.dm-0.out‚41156.23046875 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 334.23046875 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 334.23046875 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 66.8896484375 disk.nvme0n1.in‚ 92.70703125 network.recv‚818734 disk.drbd1289.in‚0 disk.drbd2111.in‚0€ì‡ŒžÉ‚(:“ ·ÝŠÓíôȳ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚255.595' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.19833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z‡Ø*?Ê‚(:´  ·ÝŠÓÛÆæ³ disk.sda.in‚0.03125 disk.rbd1.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 network.recv‚818884 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 cpu‚25.326858136765605 disk.sda2.out‚ 334.4921875 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0# disk.nvme0n1.out‚41156.45703125 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd2111.in‚0) proc.memory.availableMB‚ 2913.28515625 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.dm-0.out‚41156.45703125 disk.nvme0n1.in‚ 92.70703125 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 66.9912109375) disk./.usagePercent‚70.51355896808111 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0) proc.memory.percent‚11.81344985961914 network.sent‚ 1.2507782e+07 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1226.in‚0 disk.sda.out‚ 334.4921875 disk.drbd1931.in‚0$ memory_percent‚88.14581235249838 disk.drbd1817.in‚0 disk.drbd1673.in‚0% disk./.usageGB‚1260.8540496826172 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0R·0¨‘Ë‚(:† ÆÝŠÓÒ›Á³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚108.477' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.159 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ØÃEÌ‚(:º  ÆÝŠÓÀ«à³ disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.32680798255359 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.sda.out‚ 334.96484375 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0( disk./.usagePercent‚70.5135568347027 disk.drbd1817.out‚0 disk.drbd1931.in‚0$ memory_percent‚88.14581235249838" proc.memory.rssMB‚ 2903.2734375 disk.drbd1828.in‚0) proc.memory.percent‚11.81344985961914 disk.drbd1584.out‚0 disk.sda2.out‚ 334.96484375 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1510.out‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 2913.28515625 network.sent‚ 1.2578726e+07# disk.dm-0.out‚41156.79833984375 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 92.70703125 network.recv‚823531 disk.dm-0.in‚ 92.70703125 disk.rbd1.in‚0 disk.rbd1.out‚ 67.1396484375& disk.nvme0n1.out‚41156.79833984375% disk./.usageGB‚1260.8540115356445 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0Eë4Í/ _wandbruntime‚12940  _runtime‚12940›CZ²ÈΠtrainer/global_step‚7099 % _timestamp‚1.7848521767631197e+09 _runtime‚12940.120364162 _step‚589 & lr-AdamW/pg1‚5.911431535315193e-05 & lr-AdamW/pg2‚5.911431535315193e-05Áhå°ÍÏÇ & lr-AdamW/pg1‚5.911431535315193e-05 & lr-AdamW/pg2‚5.911431535315193e-05 trainer/global_step‚7099 % _timestamp‚1.7848521767631197e+09 _runtime‚12940.120364162 _step‚589Íz ÔM4Ð/ _wandbruntime‚12940  _runtime‚12940©kšúÑ‚P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2051771131914752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559379749304320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2050952176239800 trainer/global_step‚7099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61645914 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61645914 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61645908 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127570505 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568442683517952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559379381213184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9062878752256 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61645908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2044360964777984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127580575 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7408821700608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7397760500936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40554154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40553795 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65934661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9062934213632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 % _timestamp‚1.7848521767641861e+09 _step‚590 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2044360938604544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65924597 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568442259965440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2051769760305152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚12940.121735464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65924597 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7410167136768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2044360964777984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21632792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2051769760305152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18921008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544&8× (Ò‡P PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65934661 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2051771131914752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2044360938604544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7408821700608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2050950806200546 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2043554415738864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18921003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21633146 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61645914 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61645914 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61645908 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127570505 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568442683517952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559379381213184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9062878752256 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61645908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2044360964777984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127580575 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7408821700608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7397760500936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40554154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40553795 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65934661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9062934213632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 % _timestamp‚1.7848521767641861e+09 _step‚590 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2044360938604544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65924597 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568442259965440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2051769760305152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚12940.121735464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65924597 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7410167136768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2044360964777984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21632792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2051769760305152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18921008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2051771131914752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559379749304320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2050952176239800 trainer/global_step‚7099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127598038 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40559689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21635694 trainer/global_step‚7099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18923643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127587954 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2051230628775962 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127587954 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568522513588224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61654281 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2051231998815328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2052051063720448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21636041 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2044639847895040 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2043833217512640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7411189651968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61654281 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2052051063720448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40559337 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2043833192084672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7409844208640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7409844208640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127598038 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61654275 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672ö—Ñ{|Õ‘N NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61654275 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18923648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚559458335670272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61654281 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2052051063720448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40559337 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2043833192084672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7409844208640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7409844208640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127598038 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61654275 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127598038 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40559689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21635694 trainer/global_step‚7099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18923643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127587954 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2051230628775962 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127587954 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568522513588224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61654281 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2051231998815328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2052051063720448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21636041 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2044639847895040 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2043833217512640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7411189651968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2065766414386176 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66380549 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62064264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7461356692992 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 _runtime‚13026.865329016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62064258 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2058306403129344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21780982 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128454877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7461356692992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚572434483535360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9125415933440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2065766414386176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9125362569216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616àñŒJ ê‡P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19053008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚572434062080000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7461356692992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚572434483535360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9125415933440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2065766414386176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9125362569216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ]HDeviceStatsMonitor.on_train_batch_staÕDÃrt/requested_bytes.large_pool.freed‚2057494478999696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40834344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40833985 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2065767785995776 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2064941998772154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7448864383744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128444807 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2065766414386176 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66380549 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62064264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7461356692992 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 _runtime‚13026.865329016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62064258 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2058306403129344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21780982 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128454877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128472340 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2058585312419840 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2065221821347570 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7448540574098 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66389631 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66399709 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2066047717801472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40839560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19055643 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7462379208192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚7149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62072625 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2058585338593280 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2066046346184704 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2066047717801472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚572514313526784 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40839915 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9126659558912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128472340 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62072631 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66399709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21784267 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7462379208192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128462256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128462256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9126604104704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755Ò¯M'í‘N QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 % _timestamp‚1.7848522649827564e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2058585312419840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7461033764864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19055648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62072631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚563387653967872 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2065223191386936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2058585338593280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚563387285876736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128472340 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2058585312419840 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2065221821347570 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7448540574098 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66389631 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66399709 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2066047717801472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40839560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19055643 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7462379208192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚7149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62072625 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2058585338593280 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2066046346184704 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2066047717801472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚572514313526784 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40839915 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9126659558912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128472340 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62072631 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66399709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21784267 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7462379208192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128462256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128462256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9126604104704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299Ó~îŸç4î/ _wandbruntime‚13028  _runtime‚130284cI[ ï† %loss/total_loss‚7.828819060330109 trainer/global_step‚7149 _runtime‚13028.378475605 /loss/FRt1/detection_loss‚0.5438256006331691 0loss/FRt1/assignment_loss‚1.6726754920640652 0loss/FRt2/assignment_loss‚0.7063956685298312 3loss/FBt1/assignment_loss‚0.0016130320124787545 1loss/SRqqt1/detection_loss‚0.9775385827650821 _step‚596 3loss/SRqqt1/assignment_loss‚0.43864208087392303 2loss/SRqqt2/assignment_loss‚0.4940133194235457 1loss/FBt2/assignment_loss‚0.01286311682458881 /loss/FRt2/detection_loss‚1.3726611919804121 /loss/FBt1/detection_loss‚0.6180020108617641 /loss/FBt2/detection_loss‚0.4152743779298709 epoch‚25 % _timestamp‚1.7848522650212688e+09 1loss/SRqqt2/detection_loss‚0.5753145864313783 "E>ð‹ 3loss/SRqqt1/assignment_loss‚0.43864208087392303 2loss/SRqqt2/assignment_loss‚0.4940133194235457 1loss/FBt2/assignment_loss‚0.01286311682458881 /loss/FRt2/detection_loss‚1.3726611919804121 /loss/FBt1/detection_loss‚0.6180020108617641 /loss/FBt2/detection_loss‚0.4152743779298709 epoch‚25 % _timestamp‚1.7848522650212688e+09 1loss/SRqqt2/detection_loss‚0.5753145864313783 %loss/total_loss‚7.828819060330109 trainer/global_step‚7149 _runtime‚13028.378475605 /loss/FRt1/detection_loss‚0.5438256006331691 0loss/FRt1/assignment_loss‚1.6726754920640652 0loss/FRt2/assignment_loss‚0.7063956685298312 3loss/FBt1/assignment_loss‚0.0016130320124787545 1loss/SRqqt1/detection_loss‚0.9775385827650821 _step‚596Ô£Ãqˆ“ñ‚(:ˆ ¯ÞŠÓˆçà³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.885' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 £ç4;ò‚(:°  ¯ÞŠÓüÁ‚´ network.recv‚830955 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1285.out‚0& disk.nvme0n1.out‚41159.49853515625 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚337.8125 disk.drbd1356.in‚0% disk./.usageGB‚1260.8568687438965 disk.drbd1285.in‚0 disk.sda.out‚337.8125 disk.sda2.in‚0.03125 disk.drbd1289.in‚0) proc.memory.percent‚11.81344985961914 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0# disk.dm-0.out‚41159.49853515625 disk.drbd2078.in‚0 disk.drbd2078.out‚0 cpu‚25.326304562636867$ memory_percent‚92.10104942321777) proc.memory.availableMB‚ 1941.24609375 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1550.in‚0 disk.sda.in‚0.03125" proc.memory.rssMB‚ 2903.2734375 network.sent‚ 1.2656647e+07 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0( disk./.usagePercent‚70.5137166247455 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.rbd1.out‚ 67.6796875 disk.drbd1584.in‚0 disk.drbd1584.out‚0ˆ²sLŸó‚(:” ¾ÞŠÓӗݳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.602' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.20066666666665 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VeÕô‚(:¹  ¾ÞŠÓ‹ìü³ disk.sda2.out‚ 338.16015625" proc.memory.rssMB‚ 2903.2734375% disk./.usageGB‚1260.8568496704102 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0) proc.memory.percent‚11.81344985961914 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 67.6796875 disk.drbd1938.in‚0# disk.dm-0.out‚41159.82275390625 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1941.24609375 network.sent‚ 1.2716719e+07 disk.drbd2041.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1285.in‚0$ memory_percent‚92.10104942321777 disk.drbd1828.out‚0 disk.sda2.in‚0Ä}“Po.03125 disk.drbd2078.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 92.70703125& disk.nvme0n1.out‚41159.82275390625 disk.drbd1510.out‚0 disk.sda.out‚ 338.16015625 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 network.recv‚833622 disk.drbd1226.out‚0 disk.drbd2118.in‚0 cpu‚25.326236366448352) disk./.usagePercent‚70.51371555805629>Z­‚žõ‚(:“ ÍÞŠÓ‡±³³ gpu.0.gpu‚39.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚228.178' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.05933333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Û®Cö‚(:¸  ÍÞŠÓŸ·Ô³ disk.drbd1584.out‚0 disk.sda.out‚ 338.72265625 disk.drbd1965.in‚0& disk.nvme0n1.out‚41160.01416015625 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0# memory_percent‚92.1011765797933) proc.memory.availableMB‚ 1941.21484375 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 67.7421875 disk.sda2.out‚ 338.72265625 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0) proc.memory.percent‚11.81344985961914% disk./.usageGB‚1260.8568496704102 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0# disk.dm-0.out‚41160.01416015625 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 cpu‚25.326188509322893 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1965.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 network.sent‚ 1.2716785e+07 network.recv‚833688) disk./.usagePercent‚70.51371555805629 disk.drbd1817.in‚0 disk.drbd1226.out‚0‘²ñSž÷‚(:“ ÜÞŠÓ‹¼¸³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚276.242' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.08066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«¦òAø‚(:¶  ÜÞŠÓãÙß³ disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 338.98828125 disk.drbd1289.in‚0 cpu‚25.32612007532886 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 92.70703125# disk.dm-0.out‚41160.34326171875 disk.drbd2078.out‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125$ memory_percent‚92.10017522176108& disk.nvme0n1.out‚41160.34326171875 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0) proc.memory.percent‚11.81344985961914 network.sent‚ 1.2722543e+07 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.51371769143469 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 network.recv‚835583 disk.rbd1.out‚ 67.921875 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 338.98828125 disk.drbd1226.in‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 1941.4609375% disk./.usageGB‚1260.8568878173828§òIÛžù‚(:“ ëފӠ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.571' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.52366666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H‘`$>ú‚(:³  ëÞŠÓ†µÌ³ disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1828.in‚0 disk.drbd1552.in‚0 network.sent‚ 1.2723519e+07 network.recv‚836440 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.rbd1.out‚ 68.078125 disk.drbd1285.in‚0 disk.drbd1584.out‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd2118.in‚0 disk.drbd1817.in‚0) proc.memory.percent‚11.81344985961914 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0& disk.nvme0n1.out‚41160.58154296875% disk./.usageGB‚1260.8569068908691 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.sda2.out‚ 339.3515625$ memory_percent‚92.10119247436523# disk.dm-0.out‚41160.58154296875 cpu‚25.32605275673252( proc.memory.availableMB‚ 1941.2109375 disk.drbd1817.out‚0 disk.sda.out‚ 339.3515625 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125( disk./.usagePercent‚70.5137187581239 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd1510.in‚06dâõŸû‚(:” úÞŠÓÏù׳ gpu.0.gpu‚52.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.911' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.303666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2079763068467200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41113622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576426221457408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2078933191343762 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19185008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚567238017808384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2072251893827584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2072251867654144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129319109 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚567238385899520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚7199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66836501 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62482608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9187790582272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7512546249216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62482614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19185003 _runtime‚13113.584712611 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62482608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21928973 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2079763068467200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41113622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576426221457408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2078933191343762 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19185008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚567238017808384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2072251893827584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2072251867654144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129319109 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚567238385899520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚7199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66836501 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62482608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9187790582272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7512546249216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62482614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19185003 _runtime‚13113.584712611 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62482608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21928973 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2072530803118080 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2071713394890240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 trainer/global_step‚7199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19187643 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62490981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41119526 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2080044371882496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.6w$ó´on_train_batch_end/active_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7513568764416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129346642 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7848523517034469e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66845583 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129336558 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576505644792320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2072530803118080 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2072530776944640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19187648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2080044371882496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7512223321088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62490975 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21931522 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576506070434816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9189098169344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2079213013919178 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21931878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚13115.061043686 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66855661 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7500989068304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41119165 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2080043000265728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2071713369462272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824hÃE¼'… ‘N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62490975 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7513568764416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚567316604174336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129336558 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2072530803118080 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2071713394890240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 trainer/global_step‚7199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19187643 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62490981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41119526 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2080044371882496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7513568764416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129346642 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7848523517034469e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66845583 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129336558 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576505644792320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 _step‚599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2072530803118080 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2072530776944640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19187648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2080044371882496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7512223321088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62490975 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21931522 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576506070434816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9189098169344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2079213013919178 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21931878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _runtime‚13115.061043686 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66855661 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7500989068304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41119165 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2080043000265728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2071713369462272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361׋¨Ï€4† / _wandbruntime‚13115  _runtime‚13115c¸‡ ÿ trainer/global_step‚7199 % _timestamp‚1.7848523517407618e+09 _step‚600 1loss/SRqqt1/assignment_loss‚0.591119174203721 2loss/FBt1/assignment_loss‚0.022690945607068833 2loss/FBt2/assignment_loss‚0.026324441273442505 1loss/SRqqt1/detection_loss‚0.9482169108793622 2loss/SRqqt2/assignment_loss‚0.5017119472616939 /loss/FRt2/detection_loss‚1.3486673707883594 /loss/FBt1/detection_loss‚0.7527686038699287 /loss/FRt1/assignment_loss‚1.705625358854292 /loss/FRt2/assignment_loss‚0.590396588257221 .loss/FRt1/detection_loss‚0.535800738957381 1loss/SRqqt2/detection_loss‚0.6059834607800949 _runtime‚13115.097928335 0loss/FBt2/detection_loss‚0.40410396022143485 "loss/total_loss‚8.033409500954 epoch‚25¾QvŠ ˆ „ 2loss/SRqqt2/assignment_loss‚0.5017119472616939 /loss/FRt2/detection_loss‚1.3486673707883594 /loss/FBt1/detection_loss‚0.7527686038699287 /loss/FRt1/assignment_loss‚1.705625358854292 /loss/FRt2/assignment_loss‚0.590396588257221 .loss/FRt1/detection_loss‚0.535800738957381 1loss/SRqqt2/detection_loss‚0.6059834607800949 _runtime‚13115.097928335 0loss/FBt2/detection_loss‚0.40410396022143485 "loss/total_loss‚8.033409500954 epoch‚25 trainer/global_step‚7199 % _timestamp‚1.7848523517407618e+09 _step‚600 1loss/SRqqt1/assignment_loss‚0.591119174203721 2loss/FBt1/assignment_loss‚0.022690945607068833 2loss/FBt2/assignment_loss‚0.026324441273442505 1loss/SRqqt1/detection_loss‚0.9482169108793622ØèUƒ“‰ ‚(:ˆ ‰ßŠÓÊÁó gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.529' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÚn9Š ‚(:®  ‰ßŠÓ÷¨ä³ disk.sda2.in‚0.03125 disk.nvme0n1.in‚ 92.70703125" proc.memory.rssMB‚ 2903.2734375 network.recv‚840308 disk.drbd1285.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 340.2890625 disk.drbd2078.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.10108121236166 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 cpu‚25.32597360087618 disk.drbd1828.out‚0% disk./.usageGB‚1260.8569068908691 disk.drbd2118.in‚0 disk.dm-0.out‚41161.04296875 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚41161.04296875) proc.memory.availableMB‚ 1941.23828125 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 340.2890625 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 68.140625 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1510.in‚0) proc.memory.percent‚11.81344985961914( disk./.usagePercent‚70.5137187581239 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 network.sent‚ 1.2788832e+07 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0ôƒ¯B”‹ ‚(:‰ ˜ßŠÓ‡“±³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.361' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.787 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 |%>Œ ‚(:³  ˜ßŠÓüóÔ³ disk.drbd1226.in‚0% disk./.usageGB‚1260.8569221496582 disk.drbd1965.in‚0$ disk.nvme0n1.out‚41161.919921875 disk.sda1.out‚0 disk.drbd1510.in‚0 cpu‚25.325925536989846 disk.sda.out‚ 340.51953125 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 340.51953125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1941.1796875 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 network.recv‚840440) disk./.usagePercent‚70.51371961147527 disk.drbd1552.in‚0 disk.nvme0n1.in‚ 92.70703125 network.sent‚ 1.2789626e+07 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 68.140625 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.10131963094076 disk.drbd2118.out‚0 disk.drbd1226.out‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0) proc.memory.percent‚11.81344985961914! disk.dm-0.out‚41161.919921875 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1931.out‚07ú ž ‚(:“ §ßŠÓ±¹·³ gpu.0.gpu‚65.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚110.704' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.90133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×¥øðBŽ ‚(:·  §ßŠÓºÌ׳ disk.dm-0.in‚ 92.70703125 disk.drbd2111.out‚0 disk.sda2.out‚ 341.04296875 disk.drbd2041.out‚0 disk.drbd1828.out‚0$ memory_percent‚92.10131963094076 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 68.3603515625 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1941.1796875 disk.sda.out‚ 341.04296875 disk.drbd1289.in‚0" proc.memory.rssMB‚ 2903.2734375 cpu‚25.325839094753906% disk./.usageGB‚1260.8569412231445 disk.drbd1226.out‚0 disk.drbd1510.out‚0) proc.memory.percent‚11.81344985961914 network.sent‚ 1.2794843e+07) disk./.usagePercent‚70.51372067816447 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚41162.248046875 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1550.out‚0 network.recv‚842558 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1510.in‚0 disk.sda1.in‚0! disk.dm-0.out‚41162.248046875 disk.drbd1356.out‚0 proc.cpu.threads‚17ñž ‚(:“ ¶ßŠÓìú©³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.552' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.18400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y4üB ‚(:·  ¶ßŠÓï«Æ³ disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 92.70703125) proc.memory.percent‚11.81344985961914% disk./.usageGB‚1260.8569450378418 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 341.4609375 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 68.3603515625 disk.drbd1289.out‚0# disk.dm-0.out‚41163.00439453125 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 92.70703125" proc.memory.rssMB‚ 2903.2734375 disk.drbd2118.out‚0 disk.drbd1938.in‚0 cpu‚25.32579137735102 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0$ memory_percent‚92.10131963094076 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd2078.out‚0 network.recv‚842732) disk./.usagePercent‚70.51372089150232 disk.drbd2041.in‚0 disk.drbd2041.out‚0 network.sent‚ 1.279569e+07 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 341.4609375 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚41163.00439453125 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125( proc.memory.availableMB‚ 1941.1796875]ª•ž‘ ‚(:“ ÅߊÓ´¶Ô³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.437' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.81233333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çMu]C’ ‚(:¸  ÅߊÓµùø³ disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 341.9765625 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2041.in‚0( proc.memory.availableMB‚ 1941.3671875 disk.drbd1285.out‚0& disk.nvme0n1.out‚41163.17626953125 disk.drbd1584.out‚0# disk.dm-0.out‚41163.17626953125" proc.memory.rssMB‚ 2903.2734375 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 341.9765625 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1510.out‚0 disk.sda.in‚0.03125 network.recv‚842798 disk.drbd1828.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51371982481311 disk.drbd1356.out‚0$ memory_percent‚92.10055669148764 disk.rbd1.out‚ 68.4228515625 disk.drbd1673.in‚0 disk.drbd1965.out‚0) proc.memory.percent‚11.81344985961914 network.sent‚ 1.2795756e+07 disk.drbd1550.out‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 92.70703125 disk.drbd2041.out‚0% disk./.usageGB‚1260.8569259643555 disk.drbd1226.out‚0 disk.drbd1552.in‚0 cpu‚25.32572267310018 disk.sda1.in‚0Ã6,”“ ‚(:‰ ÔߊÓÍϳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚304.749' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.583 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l4Ÿ=” ‚(:²  ÔߊÓ¾¤ñ³ network.sent‚ 1.2796198e+07 disk.drbd1285.out‚0% disk.nvme0n1.out‚41163.8974609375 disk.drbd2118.in‚0( disk./.usagePercent‚70.5137187581239 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0) proc.memory.availableMB‚ 1941.36328125 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 342.21875 cpu‚25.32563711345537 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2903.2734375" disk.dm-0.out‚41163.8974609375 disk.drbd1965.in‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 network.recv‚843773 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.03125 disk.rbd1.out‚ 68.4228515625 disk.sda.in‚0.03125 disk.sda.out‚ 342.21875 disk.drbd1289.out‚0% disk./.usageGB‚1260.8569068908691 disk.sda1.out‚0 disk.drbd1673.out‚0$ memory_percent‚92.10057258605957) proc.memory.percent‚11.81344985961914 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 92.70703125ËЗ4• / _wandbruntime‚13200  _runtime‚13200Ê¥ÒÈ–  % _timestamp‚1.7848524367713141e+09 _runtime‚13200.129717906 _step‚601 & lr-AdamW/pg1‚5.710296819452809e-05 & lr-AdamW/pg2‚5.710296819452809e-05 trainer/global_step‚7249 ØQ¤Í— Ç _runtime‚13200.129717906 _step‚601 & lr-AdamW/pg1‚5.710296819452809e-05 & lr-AdamW/pg2‚5.710296819452809e-05 trainer/global_step‚7249 % _timestamp‚1.7848524367713141e+09ÙK¨µ$4˜ / _wandbruntime‚13200  _runtime‚13200*„7K™ ‚P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848524367720816e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_bŸM®½"atch_start/allocation.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7549727538074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2093761094157824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2086197332178944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22077422 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2092925753954624 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2085374681805264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2086197358352384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571167336105984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130193411 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7563735805440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67292453 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62900964 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580417528045056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62900958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚382 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2093761094157824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7562390369280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7551072149360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚602 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2085374656377296 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67292453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19317003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580417953694720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2093759722548224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2086197332178944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7562390369280°HN\ (š ‡P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67302517 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2093759722548224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7563735805440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130203481 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2086197358352384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9250191939072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚7249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62900958 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22077045 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67292453 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62900964 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580417528045056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62900958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚382 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2093761094157824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7562390369280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7551072149360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚602 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2085374656377296 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67292453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19317003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580417953694720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2093759722548224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2086197332178944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7562390369280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848524367720816e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7549727538074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2093761094157824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2086197332178944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22077422 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2092925753954624 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2085374681805264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2086197358352384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571167336105984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130193411 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7563735805440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41399694 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580497361434112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571245922471936 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2085653483579040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62909331 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580497780785152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571246290563072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2094041025963520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7564758320640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9251438962176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552092951112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7550748339714 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130220944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2086476241469440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9251490222080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62909325 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2094039654346752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2093204206490786 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62909331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62909325 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2093205576530152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚7249 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67301535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41400069 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚13201.604516924 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19319643 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130210860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19319648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2094041025963520 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130220944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130220944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41399694 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580497361434112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571245922471936 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2085653483579040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62909331 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580497780785152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571246290563072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMo êÐjJnitor.on_train_batch_end/active_bytes.all.allocated‚2094041025963520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7564758320640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9251438962176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552092951112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7550748339714 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130220944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2086476241469440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9251490222080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62909325 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2094039654346752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2093204206490786 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62909331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62909325 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2093205576530152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚7249 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67301535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41400069 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _runtime‚13201.604516924 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19319643 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130210860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19319648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2094041025963520 Q  ‰ 2loss/FBt1/assignment_loss‚0.010612753575507365 /loss/FBt1/detection_loss‚0.9600558371826385 epoch‚25 _step‚604 2loss/FBt2/assignment_loss‚0.045450027059559375 1loss/SRqqt1/detection_loss‚1.0077149960811116 /loss/FBt2/detection_loss‚0.4433571639418951 $ _timestamp‚1.784852438282876e+09 0loss/FRt1/assignment_loss‚1.6964191735511827 0loss/FRt2/assignment_loss‚0.7444154262542725 1loss/SRqqt2/detection_loss‚0.5519048578780211 %loss/total_loss‚8.361650251655874 trainer/global_step‚7249 3loss/SRqqt1/assignment_loss‚0.43198357710378854 2loss/SRqqt2/assignment_loss‚0.5235814976387116 /loss/FRt1/detection_loss‚0.5594867966124037 .loss/FRt2/detection_loss‚1.386668144776783 _runtime‚13201.640113664Üî$XŸ¡ ‚(:” ãߊÓÙáų gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚303.871' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.29033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®t%9¢ ‚(:®  ãߊÓ’¹â³ disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0) proc.memory.percent‚11.81344985961914 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.rbd1.out‚ 68.736328125 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.sda.in‚0.03125 disk.drbd1828.in‚0' proc.memory.availableMB‚ 1941.296875) disk./.usagePercent‚70.51371982481311 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1356.in‚0 cpu‚25.32558965805948 disk.sda2.out‚342.5625% disk.nvme0n1.out‚41164.1591796875" proc.memory.rssMB‚ 2903.2734375 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 network.recv‚843881 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0" disk.dm-0.out‚41164.1591796875 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1226.in‚0% disk./.usageGB‚1260.8569259643555 disk.drbd2041.out‚0 network.sent‚ 1.2796306e+07 disk.sda.out‚342.5625 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0$ memory_percent‚92.10084279378256 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 92.70703125 disk.drbd2111.out‚032„þ8£ ‚(:­  òߊÓÏË„´ disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0$ memory_percent‚92.10084279378256 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2903.2734375 network.sent‚ 1.2796372e+07 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 68.822265625 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0" disk.dm-0.out‚41164.5283203125 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 92.70703125) proc.memory.percent‚11.81344985961914 network.recv‚843947) disk./.usagePercent‚70.51371982481311 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 92.70703125 cpu‚25.325521956566337 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda.out‚343.125 disk.drbd1510.in‚0' proc.memory.availableMB‚ 1941.296875% disk./.usageGB‚1260.8569259643555 disk.drbd1931.out‚0 disk.drbd1550.in‚0% disk.nvme0n1.out‚41164.5283203125 disk.sda1.out‚0 disk.sda2.out‚343.125 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0è0럤 ‚(:” òߊÓÍùà³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.614' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.20466666666665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Ïž¥ ‚(:“ àŠÓò‰Á³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚283.574' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.52466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ÂÎlB¦ ‚(:·  àŠÓžŒâ³ disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0.03125" disk.dm-0.out‚41164.6884765625 network.sent‚ 1.2869431e+07 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 343.53515625 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 92.70703125 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8566703796387 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0' proc.memory.availableMB‚ 1941.296875 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 network.recv‚849029% disk.nvme0n1.out‚41164.6884765625 disk.drbd1289.out‚0 disk.dm-0.in‚ 92.70703125 disk.drbd1965.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.10084279378256) proc.memory.percent‚11.81344985961914 disk.sda1.out‚0 disk.drbd1817.in‚0" proc.memory.rssMB‚ 2903.2734375 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 343.53515625 disk.drbd2078.out‚0 disk.rbd1.out‚ 68.900390625 cpu‚25.325437708860054) disk./.usagePercent‚70.51370553117778 disk.drbd2041.in‚0ý“MŸ§ ‚(:” àŠÓ̴ʳ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚314.408' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.80266666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"a[ðD¨ ‚(:¹  àŠÓœªê³ disk.drbd1965.in‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 68.900390625 disk.drbd1931.in‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1941.046875 disk.nvme0n1.in‚ 93.33203125% disk./.usageGB‚1260.8566513061523& disk.nvme0n1.out‚41164.99462890625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.813513437906902 disk.drbd1673.in‚0$ memory_percent‚92.10186004638672 disk.drbd1552.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚41164.99462890625 network.sent‚ 1.2870237e+07 disk.drbd2111.in‚0 disk.sda.out‚ 343.81640625 disk.drbd1673.out‚0 network.recv‚849161 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 cpu‚25.32642426773349 disk.sda2.in‚0.03125 disk.drbd1938.out‚0" proc.memory.rssMB‚ 2903.2890625) disk./.usagePercent‚70.51370446448857 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 343.81640625 disk.drbd1931.out‚0 disk.dm-0.in‚ 93.33203125 "4© / _wandbruntime‚13273  _runtime‚13273’•ÞÝÉ:ª Ãu 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8317540287971497 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.32527393102645874 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3120424747467041 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39718103408813477 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6678458452224731 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7493185997009277 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13528700172901154 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27641913294792175 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5395166277885437 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7929986715316772 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24418947100639343 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7470178604125977 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.694712221622467 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05614342540502548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7659075856208801 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5884331464767456 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5665677785873413 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 *jet/accuracy_2_of_2‚0.1681528091430664 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8339518308639526 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8425324559211731 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030883366242051125 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1548262983560562 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.668714165687561 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6678458452224731 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49016350507736206 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6920624375343323 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978915452957153 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7validation_average_jet_accuracy‚0.40386444330215454 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2471766173839569 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782377302646637 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3369586169719696 'particle/f_score‚0.6558173298835754 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6181055903434753 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.32527393102645874 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05138814076781273 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28009796142578125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7419777512550354 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026602044701576233 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4998027980327606 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7356653213500977 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3956114649772644 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997065007686615 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306546568870544 *jet/accuracy_1_of_2‚0.7992945909500122 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.316451758146286 +jet/accuracy_3_of_4‚0.10461223870515823 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4541337788105011 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 +jet/accuracy_3_of_3‚0.07590913772583008 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8049241900444031 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7128399014472961 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16745176911354065 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7355118989944458 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7619013786315918 particle/accuracy_2_of_3‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028565390035510063 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 *jet/accuracy_2_of_3‚0.5961372256278992 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26722145080566406 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7333275675773621 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6603808999061584 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9993678331375122 +particle/sensitivity‚0.6436795592308044 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.709086537361145 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4467862546443939 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024945523589849472 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 % _timestamp‚1.7848525096423514e+09 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8928571343421936 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6513006091117859 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7984469532966614 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3760443329811096 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7651387453079224 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33771637082099915 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 +jet/accuracy_1_of_1‚0.36850622296333313 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6627755761146545 *jet/accuracy_1_of_3‚0.9419974684715271 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32434096932411194 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2700575292110443 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21335415542125702 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7127524018287659 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8099741339683533 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890350103378296 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8700326681137085 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9208689332008362 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28212183713912964 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6739140748977661 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _runtime‚13273.000431405 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7609258890151978 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07312167435884476 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8441557884216309 +particle/specificity‚0.9127781987190247 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37724795937538147 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.667131245136261 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14819876849651337 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11269406229257584 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26709985733032227 *jet/accuracy_2_of_4‚0.8286618590354919 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26492103934288025 trainer/global_step‚7279 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9921409487724304 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8700326681137085 particle/accuracy_3_of_4‚1 /particle/accuracy_4_of_4‚0.9685583114624023 )loss/val_total_loss‚7.998525388567207 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6475645899772644 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34292981028556824 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018884221091866493 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04229891672730446 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4608950912952423 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14545893669128418 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3424983620643616 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 epoch‚25 particle/accuracy_1_of_2‚1 _step‚605 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7508644461631775 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7405095100402832 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8723865151405334 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5975789427757263 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6199701428413391 (particle/accuracy‚0.8550443053245544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8622973561286926 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3382245600223541 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2819916307926178 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9321929216384888 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6993216872215271 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6972059607505798 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8093321919441223 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018884221091866493 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6943493485450745 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21671833097934723 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2573923170566559 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9743425250053406æÃ]=‘« Èu 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26709985733032227 *jet/accuracy_2_of_4‚0.8286618590354919 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26492103934288025 trainer/global_step‚7279 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9921409487724304 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8700326681137085 particle/accuracy_3_of_4‚1 /particle/accuracy_4_of_4‚0.9685583114624023 )loss/val_total_loss‚7.998525388567207 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6475645899772644 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34292981028556824 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018884221091866493 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04229891672730446 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4608950912952?O^=5423 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14545893669128418 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3424983620643616 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 epoch‚25 particle/accuracy_1_of_2‚1 _step‚605 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7508644461631775 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7405095100402832 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8723865151405334 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5975789427757263 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6199701428413391 (particle/accuracy‚0.8550443053245544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8622973561286926 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3382245600223541 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2819916307926178 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9321929216384888 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6993216872215271 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6972059607505798 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8093321919441223 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018884221091866493 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6943493485450745 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21671833097934723 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2573923170566559 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9743425250053406 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8317540287971497 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.32527393102645874 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3120424747467041 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39718103408813477 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6678458452224731 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7493185997009277 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13528700172901154 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27641913294792175 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5395166277885437 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7929986715316772 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24418947100639343 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7470178604125977 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.694712221622467 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05614342540502548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7659075856208801 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5884331464767456 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5665677785873413 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 *jet/accuracy_2_of_2‚0.1681528091430664 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8339518308639526 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8425324559211731 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030883366242051125 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1548262983560562 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.668714165687561 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6678458452224731 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49016350507736206 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6920624375343323 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978915452957153 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7validation_average_jet_accuracy‚0.40386444330215454 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2471766173839569 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782377302646637 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3369586169719696 'particle/f_score‚0.6558173298835754 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6181055903434753 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.32527393102645874 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05138814076781273 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28009796142578125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7419777512550354 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026602044701576233 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4998027980327606 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7356653213500977 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3956114649772644 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997065007686615 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306546568870544 *jet/accuracy_1_of_2‚0.7992945909500122 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.316451758146286 +jet/accuracy_3_of_4‚0.10461223870515823 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4541337788105011 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 +jet/accuracy_3_of_3‚0.07590913772583008 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8049241900444031 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7128399014472961 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16745176911354065 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7355118989944458 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7619013786315918 particle/accuracy_2_of_3‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028565390035510063 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 *jet/accuracy_2_of_3‚0.5961372256278992 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26722145080566406 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7333275675773621 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6603808999061584 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9993678331375122 +particle/sensitivity‚0.6436795592308044 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.709086537361145 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4467862546443939 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024945523589849472 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 % _timestamp‚1.7848525096423514e+09 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8928571343421936 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6513006091117859 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7984469532966614 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3760443329811096 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7651387453079224 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33771637082099915 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 +jet/accuracy_1_of_1‚0.36850622296333313 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6627755761146545 *jet/accuracy_1_of_3‚0.9419974684715271 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32434096932411194 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2700575292110443 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21335415542125702 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7127524018287659 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8099741339683533 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890350103378296 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8700326681137085 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9208689332008362 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28212183713912964 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6739140748977661 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _runtime‚13273.000431405 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7609258890151978 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07312167435884476 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8441557884216309 +particle/specificity‚0.9127781987190247 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37724795937538147 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.667131245136261 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14819876849651337 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11269406229257584Ý=“ˬ ‚(:’ ŸàŠÓóþ²³ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚58.0 gpu.0.powerWatts‚106.441' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.480333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i|Èá:­ ‚(:¯  ŸàŠÓåþÒ³ disk.drbd1552.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0# disk.dm-0.out‚41165.21923828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 cpu‚25.328856040533566 proc.cpu.threads‚17 network.sent‚ 1.2926177e+07 disk.drbd1289.out‚0 disk.rbd1.out‚ 69.1201171875 disk.dm-0.in‚ 93.33203125 disk.drbd2078.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 3873.80859375) disk./.usagePercent‚70.56919235686635 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0% proc.memory.availableMB‚ 336.59375% disk./.usageGB‚1261.8488311767578 disk.sda.out‚344.125 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 93.33203125& disk.nvme0n1.out‚41165.21923828125 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.sda2.in‚0.03125 network.recv‚851767 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0* proc.memory.percent‚15.762567520141602 disk.sda.in‚0.03125 disk.sda2.out‚344.125 disk.drbd1510.in‚0# memory_percent‚98.6303965250651 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0ðÝIÕŸ® ‚(:” ®àŠÓõþÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚304.164' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.38799999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~>ôã?¯ ‚(:´  ®àŠÓç û³ network.sent‚ 1.2929156e+07 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0# disk.nvme0n1.out‚42135.20703125 disk.rbd1.out‚ 69.1201171875 disk.drbd1817.in‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1673.in‚0 disk.dm-0.out‚42135.20703125 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2904.2734375* proc.memory.percent‚11.817518870035808 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.8607482910156 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0' proc.memory.availableMB‚ 2913.234375 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 cpu‚25.327846586827096 disk.sda.out‚ 344.44921875 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51393358932964 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 344.44921875 disk.drbd2078.in‚0# memory_percent‚88.1460189819336 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1817.out‚0 network.recv‚852817 disk.drbd1931.out‚0 disk.drbd1356.in‚0O¶ž° ‚(:“ ½àŠÓ„â½³ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚293.598' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.86600000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦}ï9?± ‚(:´  ½àŠÓ“÷ܳ disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 network.recv‚853785 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.drbd1226.out‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 69.1826171875 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51382436035513% disk./.usageGB‚1260.8587951660156 disk.sda.in‚0.03125 cpu‚25.327816213563388 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0" disk.nvme0n1.out‚ 42943.4140625# memory_percent‚92.1013037363688 network.sent‚ 1.2932212e+07 disk.drbd1673.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.dm-0.in‚ 93.33203125 disk.dm-0.out‚ 42943.4140625 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1941.18359375 disk.nvme0n1.in‚ 93.33203125 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 344.79296875 disk.sda2.out‚ 344.79296875 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd1552.out‚0m¯LÛ4² / _wandbruntime‚13311  _runtime‚13311S˜=Sȳ  & lr-AdamW/pg1‚5.643278512580202e-05 & lr-AdamW/pg2‚5.643278512580202e-05 trainer/global_step‚7299 % _timestamp‚1.7848525483072588e+09 _runtime‚13311.664482875 _step‚606ÈöZkÍ´ Ç & lr-AdamW/pg1‚5.643278512580202e-05 & lr-AdamW/pg2‚5.643278512580202e-05 trainer/global_step‚7299 % _timestamp‚1.7848525483072588e+09 _runtime‚13311.664482875 _step‚606ÞWgÀ4µ / _wandbruntime‚13311  _runtime‚13311ŽÆà(¶ P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2102703473428480 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131214527 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67795717 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚13311.668934595 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2110322671793152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2102703499601920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131204457 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2101873771645232 trainer/global_step‚7299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63408740 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2109481556516714 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2101873746217264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7620543800832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2110324043402752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9319784640000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63408740 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2109481556516714 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2101873746217264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7620543800832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2110324043402752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9319784jX–­:640000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2110322671793152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2102703499601920߆­Èy4¸ / _wandbruntime‚13313  _runtime‚13313/ÚcK'¹ ŒN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19458540 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584545670051328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7620220872704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7607461061836 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131231990 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63417113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚13313.142448508 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2109761379092242 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2109760009052876 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7620220872704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621566316032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575224608373760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22255965 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9321018806272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2102152573419008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575224240282624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2102152547991040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63417107 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67814877 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22256277 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7608805673234 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2102982382718976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63417113 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67804799 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41714500 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚312 trainer/global_step‚7299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2110603975208448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131231990 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078ü,zÆ'º ‘N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2109761379092242 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2109760009052876 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7620220872704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621566316032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575224608373760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22255965 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9321018806272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2102152573419008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575224240282624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2102152547991040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63417107 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67814877 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22256277 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7608805673234 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2102982382718976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63417113 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67804799 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41714500 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚312 trainer/global_step‚7299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2110603975208448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131231990 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19458540 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584545670051328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7620220872704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7607461061836 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131231990 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63417113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚13313.142448508 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544àr¿úü4» / _wandbruntime‚13313  _runtime‚13313¨žÅ! ¼ „ _step‚609 1loss/FBt2/assignment_loss‚0.01239447689412874 /loss/FRt1/detection_loss‚0.5349317875216201 0loss/FBt1/detection_loss‚0.39008900319988077 /loss/FBt2/detection_loss‚0.3982547877323149 trainer/global_step‚7299 % _timestamp‚1.7848525498191319e+09 0loss/FRt1/assignment_loss‚1.6561479232538765 3loss/SRqqt1/assignment_loss‚0.49700158340131717 2loss/SRqqt2/assignment_loss‚0.4575320742262226 .loss/FRt2/detection_loss‚1.348012278250795 1loss/SRqqt2/detection_loss‚0.6263136684190965 _runtime‚13313.176333176 0loss/FRt2/assignment_loss‚0.7041653382017258 3loss/FBt1/assignment_loss‚0.0003759061705100101 0loss/SRqqt1/detection_loss‚0.920822326875201 $loss/total_loss‚7.54604115414669 epoch‚26.4i+½ ‰ 0loss/FBt1/detection_loss‚0.39008900319988077 /loss/FBt2/detection_loss‚0.3982547877323149 trainer/global_step‚7299 % _timestamp‚1.7848525498191319e+09 0loss/FRt1/assignment_loss‚1.6561479232538765 3loss/SRqqt1/assignment_loss‚0.49700158340131717 2loss/SRqqt2/assignment_loss‚0.4575320742262226 .loss/FRt2/detection_loss‚1.348012278250795 1loss/SRqqt2/detection_loss‚0.6263136684190965 _runtime‚13313.176333176 0loss/FRt2/assignment_loss‚0.7041653382017258 3loss/FBt1/assignment_loss‚0.0003759061705100101 0loss/SRqqt1/detection_loss‚0.920822326875201 $loss/total_loss‚7.54604115414669 epoch‚26 _step‚609 1loss/FBt2/assignment_loss‚0.01239447689412874 /loss/FRt1/detection_loss‚0.5349317875216201á¸E¾ ‚(:º  ÌàŠÓ÷ªà³$ memory_percent‚92.10055669148764% disk./.usageGB‚1260.8587341308594 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 345.1328125 disk.drbd1938.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 network.recv‚853851 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0( proc.memory.availableMB‚ 1941.3671875 disk.drbd1673.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚43105.86767578125 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd2041.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚43105.86767578125 disk.drbd1817.out‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51382094694966 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0" proc.memory.rssMB‚ 2904.2734375 cpu‚25.327728703085693 network.sent‚ 1.2932278e+07 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.sda2.out‚ 345.1328125 disk.drbd2111.out‚0 disk.rbd1.out‚ 69.1826171875 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0È`cÕž¿ ‚(:“ ÌàŠÓ츼³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.937' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.97900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B]9ãžÀ ‚(:“ ÛàŠÓ­Ý«³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚281.685' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.89500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s”å’NÁ ‚(:·  ÛàŠÓÓþͳ disk.drbd1584.out‚0) disk./.usagePercent‚70.51381774688205 disk.rbd1.out‚ 69.31640625 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚43106.25048828125 disk.rbd1.in‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda2.out‚ 345.4609375 disk.drbd1931.in‚0 disk.ý|¾Þôsda1.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 1941.2734375 network.sent‚ 1.2932762e+07% disk./.usageGB‚1260.8586769104004 disk.drbd2078.in‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1285.in‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0# memory_percent‚92.1009381612142 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚43106.25048828125 disk.drbd1356.in‚0 disk.drbd1289.out‚0 cpu‚25.327716863800717 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 345.4609375* proc.memory.percent‚11.817518870035808 network.recv‚854770 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1552.out‚0 disk.drbd1510.out‚0îØ ÏŸ ‚(:” êàŠÓî¬Ì³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚304.301' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.43366666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýO4à ‚(:©  êàŠÓ½é³ disk.drbd1552.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0! memory_percent‚92.10205078125 network.sent‚ 1.2932828e+07 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚43106.642578125 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 345.76953125* proc.memory.percent‚11.817518870035808 disk.drbd1226.out‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2904.2734375) disk./.usagePercent‚70.51382201363887 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 345.76953125! disk.dm-0.out‚43106.642578125 disk.drbd1931.out‚0 proc.memory.availableMB‚1941 cpu‚25.3276475774706% disk./.usageGB‚1260.8587532043457 disk.drbd2078.out‚0 disk.rbd1.out‚ 69.49609375 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 network.recv‚854836 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0®. ¤9Ä ‚(:®  ùàŠÓ¾—Õ³ disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1965.in‚0 disk.drbd2111.out‚0$ disk./.usageGB‚1260.858787536621 disk.drbd1285.in‚0$ disk.nvme0n1.out‚43106.869140625 disk.drbd1226.in‚0 disk.sda.out‚346.4375 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.817518870035808 cpu‚25.327561271031424 network.sent‚ 1.3001067e+07 disk.drbd1584.out‚0! disk.dm-0.out‚43106.869140625 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚346.4375 network.recv‚859767) proc.memory.availableMB‚ 1940.99609375) disk./.usagePercent‚70.51382393367945 disk.drbd2078.out‚0$ memory_percent‚92.10206667582193 disk.rbd1.out‚ 69.55859375 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0ñÌ€ó“Å ‚(:ˆ ùàŠÓ׉µ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚283.758' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.586 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ŠŸÆ ‚(:” ˆáŠÓױ᳠gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚301.058' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.35266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!æ\?Ç ‚(:´  ˆáŠÓªÈ„´ disk.dm-0.in‚ 93.33203125 disk.drbd1828.out‚0 disk.drbd1226.out‚0# disk.dm-0.out‚43107.17626953125 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1940.99609375 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51382393367945 disk.sda2.out‚ 346.71875 disk.rbd1.in‚0 disk.rbd1.out‚ 69.55859375 disk.drbd2118.in‚0 network.sent‚ 1.3003549e+07 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd2111.in‚0 disk.sda1.in‚0$ disk./.usageGB‚1260.858787536621 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1356.in‚0 network.recv‚860720 disk.sda2.in‚0.03125 disk.drbd2041.in‚0& disk.nvme0n1.out‚43107.17626953125 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1584.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 346.71875$ memory_percent‚92.10206667582193 disk.drbd1673.out‚0 disk.drbd1289.out‚0 cpu‚25.3274917090881041Ý"¿“È ‚(:ˆ —áŠÓɨå³ gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.204' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.068 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'í\/HÉ ‚(:½  —áŠÓ” „´ disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 69.6923828125 disk.sda2.out‚ 347.01953125 disk.drbd1356.out‚0% disk./.usageGB‚1260.8588066101074 disk.drbd1356.in‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1940.99609375 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0" proc.memory.rssMB‚ 2904.2734375 network.sent‚ 1.3004515e+07 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.817518870035808 disk.sda2.in‚0.03125 disk.drbd1510.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51382500036864 disk.nvme0n1.in‚ 93.33203125 disk.drbd1510.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0$ memory_percent‚92.10206667582193 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 93.33203125 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚43107.35205078125 disk.drbd1965.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 cpu‚25.327442396043995 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 347.01953125 disk.drbd1828.in‚0& disk.nvme0n1.out‚43107.35205078125 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 network.recv‚861587·¬ç4Ê / _wandbruntime‚13398  _runtime‚13398FWÈË Â & lr-AdamW/pg2‚5.576288147756592e-05 trainer/global_step‚7349 % _timestamp‚1.7848526349830256e+09 _runtime‚13398.340237942 _step‚610 & lr-AdamW/pg1‚5.576288147756592e-05…a+ÍÌ Ç _runtime‚13398.340237942 _step‚610 & lr-AdamW/pg1‚5.576288147756592e-05 & lr-AdamW/pg2‚5.576288147756592e-05 trainer/global_step‚7349 % _timestamp‚1.7848526349830256e+09â/g‡ü4Í / _wandbruntime‚13398  _runtime‚13398—‘ˆ(Î €P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19587900 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7657544143004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132088829 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2124319325874176 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2115813860334032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68251669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚579075340305408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 % _timestamp‚1.7848526349840322e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68261733 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41989774 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588457578761216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2116648964126720 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63827096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22402227 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19587895 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132078759 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2116648964126720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9382662008320 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2116648937953280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚353 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2124319325874176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22401879 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63827090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68261733 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588458002313728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63827096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2124320697483776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63827090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7671733357056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2116648937953280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7671733357056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9382606546944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181¼C (Ï …P cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9382662008320 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2116648937953280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚353 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2124319325874176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22401879 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63827090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68261733 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588458002313728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63827096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2124320697483776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63827090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7671733357056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2116648937953280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7671733357056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9382606546944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19587900 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7657544143004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132088829 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2124319325874176 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2115813860334032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68251669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚579075340305408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 % _timestamp‚1.7848526349840322e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68261733 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41989774 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588457578761216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2116648964126720 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63827096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22402227 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19587895 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132078759 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2116648964126720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544ã\g˲4Ð / _wandbruntime‚13399  _runtime‚13399m’ÓQû Ñ ŒN @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63835463 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2116927847243776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68260751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41995344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22404809 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588537408957440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚13399.816461673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132096208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63835457 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2116092662107808 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2124600629289472 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2116927873417216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132106292 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123751201624484 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2124600629289472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68270829 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2123752571663850 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7671410428928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132106292 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2124599257672704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63835463 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚579153558580224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2116927873417216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7658564944644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7671410428928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132096208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7672755872256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9383910025728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚7349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63835457 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19590540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7672755872256 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 % _timestamp‚1.7848526364588795e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68270829 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9383850377216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41995695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚579153926671360ö•s.'Ò ‘N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132106292 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2124599257672704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63835463 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚579153558580224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2116927873417216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7658564944644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7671410428928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132096208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7672755872256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9383910025728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚7349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63835457 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19590540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7672755872256 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 % _timestamp‚1.7848526364588795e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68270829 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9383850377216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41995695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚579153926671360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63835463 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2116927847243776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68260751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41995344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22404809 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588537408957440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚13399.816461673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132096208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63835457 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2116092662107808 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2124600629289472 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2116927873417216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132106292 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123751201624484 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2124600629289472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68270829 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2123752571663850 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7671410428928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653ä¨uù74Ó / _wandbruntime‚13399  _runtime‚13399½° Ô … 2loss/SRqqt2/assignment_loss‚0.4988756614310263 1loss/FBt2/assignment_loss‚0.01550625312679928 1loss/SRqqt2/detection_loss‚0.6100484978247984 /loss/FBt1/detection_loss‚0.7738984432635885 _runtime‚13399.853288471 _step‚613 0loss/FRt1/assignment_loss‚1.6174988143429727 0loss/FRt2/assignment_loss‚0.7038234579912965 0loss/FBt2/detection_loss‚0.42627875061431364 2loss/SRqqt1/assignment_loss‚0.5606356283946267 1loss/SRqqt1/detection_loss‚0.9010577182013194 %loss/total_loss‚8.011901601996549 epoch‚26 trainer/global_step‚7349 3loss/FBt1/assignment_loss‚0.0011280655287912327 /loss/FRt1/detection_loss‚0.5568942050975125 .loss/FRt2/detection_loss‚1.346256106179502 % _timestamp‚1.7848526364960923e+09Bâ/ÙÕ Š 3loss/FBt1/assignment_loss‚0.0011280655287912327 /loss/FRt1/detection_loss‚0.5568942050975125 .loss/FRt2/detection_loss‚1.346256106179502 % _timestamp‚1.7848526364960923e+09 2loss/SRqqt2/assignment_loss‚0.4988756614310263 1loss/FBt2/assignment_loss‚0.01550625312679928 1loss/SRqqt2/detection_loss‚0.6100484978247984 /loss/FBt1/detection_loss‚0.7738984432635885 _runtime‚13399.853288471 _step‚613 0loss/FRt1/assignment_loss‚1.6174988143429727 0loss/FRt2/assignment_loss‚0.7038234579912965 0loss/FBt2/detection_loss‚0.42627875061431364 2loss/SRqqt1/assignment_loss‚0.5606356283946267 1loss/SRqqt1/detection_loss‚0.9010577182013194 %loss/total_loss‚8.011901601996549 epoch‚26 trainer/global_step‚7349å¼¼ÓžÖ ‚(:“ ¦áŠÓËÛ¼³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.406' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.13533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©XÞ¿B× ‚(:·  ¦áŠÓр߳ disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.nvme0n1.in‚ 93.33203125& disk.nvme0n1.out‚43107.78662109375 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 cpu‚25.327357138562594 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 347.265625 disk.rbd1.out‚ 69.6923828125 disk.drbd1226.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 network.recv‚862584 disk.sda2.out‚ 347.265625 disk.drbd1552.in‚0 disk.drbd1356.in‚0 network.sent‚ 1.30075e+07 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0$ memory_percent‚92.10219383239746% disk./.usageGB‚1260.8588066101074 disk.drbd1673.in‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1285.out‚0" proc.memory.rssMB‚ 2904.2734375) disk./.usagePercent‚70.51382500036864 disk.sda1.in‚0 disk.drbd2111.in‚0 disk.sda.in‚0.03125# disk.dm-0.out‚43107.78662109375 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1940.96484375|u%BžØ ‚(:“ µáŠÓÄÛÙ³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.096' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.36533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý22AÙ ‚(:¶  µáŠÓÖÄõ³ disk.drbd1226.in‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1817.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚43107.98583984375% disk./.usageGB‚1260.8588104248047 disk.drbd1584.in‚0 disk.drbd2041.out‚0( disk./.usagePercent‚70.5138252137065 disk.drbd2111.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd1285.in‚0$ memory_percent‚92.10507074991861" proc.memory.rssMB‚ 2904.2734375 network.sent‚ 1.3067832e+07 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚43107.98583984375 disk.drbd1938.out‚0 disk.rbd1.out‚ 69.7548828125 network.recv‚864408 disk.drbd1817.in‚0 disk.sda2.out‚ 347.890625 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1828.in‚0 cpu‚25.32732595903038 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 347.890625 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1940.2578125º>ážÚ ‚(:“ ÄáŠÓÿÅ°³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚222.183' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.06099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06K›?Û ‚(:´  ÄáŠÓ£‡Ò³ disk.drbd1550.out‚0# memory_percent‚92.1051025390625 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd2078.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚43108.85107421875 disk.drbd1356.in‚0 disk.rbd1.out‚ 69.7548828125 disk.drbd1828.in‚0 disk.drbd1828.out‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1584.out‚0 network.sent‚ 1.3067898e+07 network.recv‚864474 disk.drbd1817.in‚0 disk.sda2.out‚ 348.33984375 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚43108.85107421875 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 cpu‚25.32727806584934 disk.drbd2078.in‚0# proc.memory.availableMB‚1940.25 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.sda.out‚ 348.33984375 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1356.out‚0$ disk./.usageGB‚1260.858741760254 disk.sda.in‚0.03125 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd2118.in‚0* proc.memory.percent‚11.817518870035808 proc.cpu.threads‚17) disk./.usagePercent‚70.51382137362535 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1285.in‚0ÀI'žÜ ‚(:“ ÓáŠÓ¢Ä³³ gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.502' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.50066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06“hEÝ ‚(:º  ÓáŠÓáºÑ³) disk./.usagePercent‚70.51382244031456 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.drbd1817.out‚0 disk.drbd1226.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2904.2734375 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.817518870035808 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 69.888671875 disk.drbd1356.out‚0% disk./.usageGB‚1260.8587608337402 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0# memory_percent‚92.1060880025228 disk.drbd2118.in‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 93.33203125 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 348.70703125 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 348.70703125 disk.drbd1938.in‚0 network.recv‚866751 disk.sda2.in‚0.03125# disk.dm-0.out‚43109.03857421875( proc.memory.availableMB‚ 1940.0078125 cpu‚25.327192152789753 proc.cpu.threads‚17& disk.nvme0n1.out‚43109.03857421875 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 network.sent‚ 1.3076062e+07ËDì.œÞ ‚(:‘ âáŠÓø«³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.664' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.55466666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uë^½<ß ‚(:±  âáŠÓո̳ disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8588104248047 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd2078.out‚0& disk.nvme0n1.out‚43109.55615234375 disk.drbd2118.in‚0 cpu‚25.327125072939875 disk.drbd1828.in‚0 disk.drbd1828.out‚0# disk.dm-0.out‚43109.55615234375 disk.drbd1356.out‚0* proc.memory.percent‚11.817518870035808 network.recv‚866859 disk.drbd1817.in‚0 disk.drbd1552.out‚0 network.sent‚ 1.307617e+07 disk.drbd1510.out‚0" proc.memory.rssMB‚ 2904.2734375 disk.sda.out‚ 348.9609375 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 93.33203125 disk.sda2.out‚ 348.9609375 disk.drbd2111.out‚0( disk./.usagePercent‚70.5138252137065 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda1.out‚0# memory_percent‚92.1051025390625 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 69.974609375 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.03125# proc.memory.availableMB‚1940.25 disk.drbd1289.in‚0 disk.drbd1965.in‚0³ÍéŸà ‚(:” ñáŠÓÿð°³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚307.721' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.57366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y€]Ý>á ‚(:³  ñáŠÓ°àѳ disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 349.66796875 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚70.5138252137065 disk.rbd1.out‚ 70.037109375 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0# proc.memory.availableMB‚1940.25 disk.drbd1510.in‚0 network.recv‚867801 disk.sda2.out‚ 349.66796875 disk.dm-0.in‚ 93.33203125# disk.dm-0.out‚43109.76318359375 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.sda.in‚0.03125" proc.memory.rssMB‚ 2904.2734375* proc.memory.percent‚11.817518870035808 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 93.33203125& disk.nvme0n1.out‚43109.76318359375 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 network.sent‚ 1.3077144e+07 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2118.out‚0 cpu‚25.32705782287545% disk./.usageGB‚1260.8588104248047 disk.drbd1550.out‚0 disk.drbd2041.out‚0# memory_percent‚92.1051025390625z£€4â / _wandbruntime‚13485  _runtime‚13485™t¯zÊã Ä _runtime‚13485.315253082 _step‚614 ' lr-AdamW/pg1‚5.5093345058839814e-05 ' lr-AdamW/pg2‚5.5093345058839814e-05 trainer/global_step‚7399 % _timestamp‚1.7848527219580226e+09ìÍ7‡Ïä É ' lr-AdamW/pg1‚5.5093345058839814e-05 ' lr-AdamW/pg2‚5.5093345058839814e-05 trainer/global_step‚7399 % _timestamp‚1.7848527219580226e+09 _runtime‚13485.315253082 _step‚614æ@Έ›4å / _wandbruntime‚13485  _runtime‚13485@:p´æ P trainer/global_step‚7399 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7721577477120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64245440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68717685 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42269723 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2130594402478080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132963131 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7709992637098 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStau˜tS$tsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2130594402478080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42270080 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68707621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592449864495104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592449445136896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2129753923594864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132953061 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2137463941659930 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7722922913280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7708648025812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68717685 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68707621 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64245446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2137462571620676 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9445154625024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583004290511872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19719900 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2138315979955200 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583004290511872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19719900 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2138315979955200 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592449864495104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592449445136896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2129753923594864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132953061 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2137463941659930 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7722922913280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7708648025812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68717685 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68707621癯úÊ4è / _wandbruntime‚13486  _runtime‚13486»±m4'é ŒN F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2138595911753728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132980594 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132970510 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚351 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7723945428480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22553106 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9446459634688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9446406277632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7711013438850 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132970510 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68716703 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592529283155456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7723945428480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848527234353044e+09 _runtime‚13486.792922055 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2138597283370496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64253807 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583083244968960 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42275290 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42275646 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2130873311768576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64253813 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2137743764235458 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7709668827452 trainer/global_step‚7399 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2138597283370496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592529704603648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132980594 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 _step‚616 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68716703 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592529283155456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7723945428480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848527234353044e+09 _runtime‚13486.792922055 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2138597283370496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64253807 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583083244968960 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonittÖÖ—ßor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42275290 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42275646 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2130873311768576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64253813 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2137743764235458 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7709668827452 trainer/global_step‚7399 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2138597283370496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592529704603648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132980594 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 _step‚616 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7723945428480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22553106 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9446459634688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9446406277632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7711013438850 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356èm½ÈO4ë / _wandbruntime‚13486  _runtime‚13486ÊÓá ì † %loss/total_loss‚8.201712388464875 /loss/FRt1/assignment_loss‚1.665542154951194 4loss/FBt1/assignment_loss‚0.00034946288556139243 2loss/FBt2/assignment_loss‚0.009562027055873208 1loss/SRqqt2/detection_loss‚0.6248186617722097 % _timestamp‚1.7848527234716232e+09 3loss/SRqqt2/assignment_loss‚0.48394747418694833 /loss/FBt1/detection_loss‚0.7900297849586135 epoch‚26 /loss/FRt2/assignment_loss‚0.731280987282888 2loss/SRqqt1/assignment_loss‚0.6593280214264159 /loss/FRt2/detection_loss‚1.3853954136938675 1loss/SRqqt1/detection_loss‚0.8193491038654802 trainer/global_step‚7399 _step‚617 /loss/FRt1/detection_loss‚0.6077468385089044 /loss/FBt2/detection_loss‚0.4243624578769186 _runtime‚13486.828828055Õêëýí ‹ _step‚617 /loss/FRt1/detection_loss‚0.6077468385089044 /loss/FBt2/detection_loss‚0.4243624578769186 _runtime‚13486.828828055 %loss/total_loss‚8.201712388464875 /loss/FRt1/assignment_loss‚1.665542154951194 4loss/FBt1/assignment_loss‚0.00034946288556139243 2loss/FBt2/assignment_loss‚0.009562027055873208 1loss/SRqqt2/detection_loss‚0.6248186617722097 % _timestamp‚1.7848527234716232e+09 3loss/SRqqt2/assignment_loss‚0.48394747418694833 /loss/FBt1/detection_loss‚0.7900297849586135 epoch‚26 /loss/FRt2/assignment_loss‚0.731280987282888 2loss/SRqqt1/assignment_loss‚0.6593280214264159 /loss/FRt2/detection_loss‚1.3853954136938675 1loss/SRqqt1/detection_loss‚0.8193491038654802 trainer/global_step‚7399éÖ{%žî ‚(:“ €âŠÓÎÿ¾³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.081' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.69366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pm‡?ï ‚(:´  €âŠÓ¯ïß³ disk.drbd1584.out‚0 disk.sda.out‚ 349.9453125 disk.dm-0.in‚ 93.33203125 disk.sda.in‚0.03125 disk.drbd1285.in‚0 disk.drbd1510.out‚0% proc.memory.availableMB‚ 1940.1875 disk.drbd1550.in‚0# disk.dm-0.out‚43110.77197265625 disk.drbd1965.out‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 349.9453125 network.sent‚ 1.307721e+07 network.recv‚867867 disk.drbd1289.in‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51382414701727 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.rbd1.out‚ 70.216796875 cpu‚25.32699045822755 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8587913513184 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2078.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1552.out‚0& disk.nvme0n1.out‚43110.77197265625$ memory_percent‚92.10535685221353O«‚k“ð ‚(:ˆ âŠÓ¤”Õ³ gpu.0.gpu‚41.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.347' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.449 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ߧ1Añ ‚(:¶  âŠÓÚ¨ñ³ disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0$ memory_percent‚92.10535685221353 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.drbd1510.out‚0 network.recv‚872595) disk./.usagePercent‚70.51382201363887 disk.sda.out‚ 350.2265625 disk.drbd1931.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚43111.13916015625 disk.dm-0.in‚ 93.33203125 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚43111.13916015625 cpu‚25.326923145181407 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 70.380859375 disk.sda2.out‚ 350.2265625 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0% proc.memory.availableMB‚ 1940.1875* proc.memory.percent‚11.817518870035808% disk./.usageGB‚1260.8587532043457 disk.drbd2118.out‚0 disk.drbd1828.in‚0 network.sent‚ 1.3144966e+07 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0¯E„‰?ò ‚(:´  žâŠÓ™†Ø³ disk.drbd1673.in‚0 disk.drbd1550.in‚0 cpu‚25.326857279424143% disk./.usageGB‚1260.8587532043457 disk.drbd1584.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 network.sent‚ 1.3145761e+07 disk.drbd1931.out‚0% proc.memory.availableMB‚ 1940.1875 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2904.2734375 disk.sda2.in‚0.03125 disk.dm-0.in‚ 93.33203125 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.10535685221353 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚43111.6943359375 disk.sda2.out‚ 350.4921875 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.rbd1.out‚ 70.466796875 disk.drbd1510.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd1965.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 350.4921875 disk.sda1.out‚0 network.recv‚872727) disk./.usagePercent‚70.51382201363887 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0" disk.dm-0.out‚43111.6943359375 disk.drbd1510.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17KdÆ?žó ‚(:“ žâŠÓçÖ¹³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.761' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.25366666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò¹4}=ô ‚(:²  ­âŠÓ’ŸÙ³ disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0" disk.dm-0.out‚43111.8623046875 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 351.015625 network.sent‚ 1.3145827e+07 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0& proc.memory.availableMB‚ 1940.46875" proc.memory.rssMB‚ 2904.2734375 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51382308032808 disk.drbd2078.out‚0% disk.nvme0n1.out‚43111.8623046875 disk.sda1.out‚0 disk.drbd1285.out‚0$ disk./.usageGB‚1260.858772277832 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 93.33203125 cpu‚25.326790746892893 disk.drbd1931.in‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 351.015625$ memory_percent‚92.10421244303384 network.recv‚872793 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125* proc.memory.percent‚11.817518870035808 disk.drbd2041.out‚0 disk.rbd1.out‚ 70.529296875 disk.drbd1828.in‚0 disk.sda2.in‚0.03125³"ùíœõ ‚(:‘ ­âŠÓ÷ÑçÏ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚284.7' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.89999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&^ ôŸö ‚(:” ¼âŠÓ¦éȳ gpu.0.gpu‚52.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.167' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.72233333333331 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž|®=÷ ‚(:²  ¼âŠÓ£ê³ disk.drbd1285.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2111.in‚0% disk./.usageGB‚1260.8576011657715 disk.drbd1938.out‚0 disk.rbd1.out‚ 70.529296875 disk.drbd1285.out‚0$ memory_percent‚92.10319519042969 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0& proc.memory.availableMB‚ 1940.71875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 network.sent‚ 1.3145935e+07 network.recv‚872901 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 351.34765625 disk.dm-0.out‚43112.26953125 disk.drbd1510.in‚0 cpu‚25.32676066406123 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 351.34765625 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1226.in‚0* proc.memory.percent‚11.817518870035808 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0# disk.nvme0n1.out‚43112.26953125 disk.drbd1931.in‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd2118.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51375758561093 disk.drbd1817.out‚0 disk.drbd2078.in‚0õ9>é4ø / _wandbruntime‚13571  _runtime‚13571ídçÈù  trainer/global_step‚7449 % _timestamp‚1.7848528085101635e+09 _runtime‚13571.867400272 _step‚618 & lr-AdamW/pg1‚5.442426363050833e-05 & lr-AdamW/pg2‚5.442426363050833e-05Ž"?Íú Ç _runtime‚13571.867400272 _step‚618 & lr-AdamW/pg1‚5.442426363050833e-05 & lr-AdamW/pg2‚5.442426363050833e-05 trainer/global_step‚7449 % _timestamp‚1.7848528085101635e+09ê+ l4û / _wandbruntime‚13571  _runtime‚13571.iÈ(ü ‚P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2152314005645824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2144539893176320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586933976900608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69163573 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7772767033344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7761096519906 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7774112469504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19851900 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69163573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22697489 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2152312634036224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2143694012283664 _step‚619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42549753 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848528085116715e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7759751908620 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64663796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586933608809472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69173637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133827363 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64663790 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42549384 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2144539867002880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2152312634036224 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69163573 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7772767033344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7761096519906 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7774112469504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19851900 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69163573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22697489 L–»ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2152312634036224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2143694012283664 _step‚619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42549753 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 % _timestamp‚1.7848528085116715e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7759751908620 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64663796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586933608809472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69173637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133827363 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64663790 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42549384 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2144539867002880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2152312634036224 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42554960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2144818776293376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587012563266560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587012195175424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7760772710260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133844812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133854896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2151734956807066 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2143972839485408 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2152593937451520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64672163 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69172655 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚13573.342915107 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69182733 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2152592565834752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2144818776293376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2144818802466816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64672157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9509066908672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2151733586767700 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7762117321658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19854535 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133854896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7775134984704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133844812 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 140839168063G'€!‘N REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69182733 _step‚620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69172655 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚13573.342915107 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69182733 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2152592565834752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2144818776293376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2144818802466816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64672157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9509066908672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2151733586767700 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7762117321658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19854535 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133854896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7775134984704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133844812 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7773789541376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848528099853394e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22700425 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2143972814057440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7773789541376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19854540 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚596521262084096 trainer/global_step‚7449 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42554960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2144818776293376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587012563266560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587012195175424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7760772710260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133844812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133854896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2151734956807066 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2143972839485408 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2152593937451520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64672163 BŒ!‚(:³  ‡ãŠÓòԳ disk.drbd2041.in‚0 disk.rbd1.out‚ 70.859375 disk.drbd1673.out‚0 disk.drbd1828.in‚0$ memory_percent‚92.10427602132162 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 network.sent‚ 1.3261766e+07 disk.sda2.out‚ 352.85546875 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 cpu‚25.326395017819596 network.recv‚882623 disk.drbd1817.in‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1552.out‚0% disk./.usageGB‚1260.8617553710938 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda.out‚ 352.85546875! disk.dm-0.out‚43113.572265625 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 93.33203125 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.817518870035808 disk.drbd2111.in‚0 disk.drbd1828.out‚0$ disk.nvme0n1.out‚43113.572265625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51398991051963 disk.sda2.in‚0.03125' proc.memory.availableMB‚ 1940.453125¤ õÍž!‚(:“ ‡ãŠÓùÜ·³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.584' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.86133333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n|é&žŽ!‚(:“ –ãŠÓ̽޳ gpu.0.gpu‚78.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚157.481' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚52.49366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûï‚ @!‚(:µ  –ãŠÓç‘ÿ³ proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 cpu‚25.326309970901168 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚43113.91455078125* proc.memory.percent‚11.817518870035808% disk./.usageGB‚1260.8617362976074 disk.drbd1285.out‚0& disk.nvme0n1.out‚43113.91455078125 disk.drbd1584.out‚0" proc.memory.rssMB‚ 2904.2734375 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.dm-0.in‚ 93.33203125 disk.rbd1.out‚ 70.859375 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0$ memory_percent‚92.10527737935385 network.sent‚ 1.3264796e+07 disk.sda2.in‚0.03125 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 network.recv‚883572 disk.drbd1673.out‚0 disk.sda2.out‚ 353.234375 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.51398884383042 disk.drbd2078.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 353.234375 disk.drbd1931.out‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1940.20703125 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1584.in‚0 disk.drbd1931.in‚0é¹D]4!/ _wandbruntime‚13658  _runtime‚13658ÐCøáÈ‘! trainer/global_step‚7499 % _timestamp‚1.7848528955530527e+09 _runtime‚13658.910833878 _step‚622 & lr-AdamW/pg1‚5.375572489381732e-05 & lr-AdamW/pg2‚5.375572489381732e-05´6ÊÍ’!Ç % _timestamp‚1.7848528955530527e+09 _runtime‚13658.910833878 _step‚622 & lr-AdamW/pg1‚5.375572489381732e-05 & lr-AdamW/pg2‚5.375572489381732e-05 trainer/global_step‚7499î«vØ4“!/ _wandbruntime‚13658  _runtime‚13658ôƒ!•(”!‚P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2158485331527680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65082140 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚353 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7825302025728 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2166309288117248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22846048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7812200402714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2158485357701120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42829948 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚7499 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65082140 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69619525 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2166310659726848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22845700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7823956589568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590862927107072 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2165444956763892 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2165446326803146 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 % _timestamp‚1.7848528955538611e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2158485357701120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65082146 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19983895 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134701665 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825302025728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7823956589568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69629589 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _runtime‚13658.911339466 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7823956589568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69629589 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _runtime‚13658.911339466 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2166309288117248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22846048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7812200402714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2158485357701120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42829948 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 trainer/global_step‚7499 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65082140 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69619525 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2166310659726848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22845700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ï‡å-µ4–!/ _wandbruntime‚13660  _runtime‚13660Ã3—!‹N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 $ _timestamp‚1.784852897029615e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134719114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7811876593068 trainer/global_step‚7499 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590941881564160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134729198 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65090507 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2157912902746240 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2166589219915776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65090513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2158764240818176 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134729198 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2165726149378674 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2158764240818176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19986540 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚600513622026240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69628607 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚600513204772352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005°‰aþ EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826324540928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7824979097600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42835166 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9571691299328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2157912928174208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2166590591532544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9571740462080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2166590591532544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42835514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19986535 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134719114 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158764266991616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590941513473024 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2165724779339308 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65090513 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826324540928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7824979097600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42835166 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9571691299328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2157912928174208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2166590591532544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9571740462080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2166590591532544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42835514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19986535 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134719114 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158764266991616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590941513473024 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2165724779339308 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65090513 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590941881564160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134729198 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65090507 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2157912902746240 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2166589219915776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65090513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2158764240818176 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134729198 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2165726149378674 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2158764240818176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19986540 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚600513622026240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69628607 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚600513204772352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65500496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65500496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20115895 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9633147601920 60DeviceStatsMonitor.on_train_batckÀT( h_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 trainer/global_step‚7549 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2172430796052480 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70075477 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2180307313807872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2172430822225920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20115900 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7848529822395828e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚594792245404672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43109376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70085541 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70075477 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70085541 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135586037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22993828 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135575967 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚13745.597064305 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2171574215089232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2180305942198272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7876491581952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604425761097728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2172430822225920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43109728 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2179437519374754 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65500496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65500496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20115895 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9633147601920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 trainer/global_step‚7549 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2172430796052480 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70075477 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2180307313807872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2172430822225920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20115900 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7848529822395828e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚594792245404672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43109376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70085541 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70075477 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70085541 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135586037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22993828 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135575967 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚13745.597064305 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2171574215089232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2180305942198272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7876491581952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604425761097728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2172430822225920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43109728 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2179437519374754 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43114956 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7549 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65508857 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135593416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70084559 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2179717341950282 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7864325087274 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135603500 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2179715971910916 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7862980475876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135603500 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7876168653824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43115306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20118535 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20118540 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594870831770624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2180587245613568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7877514097152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70094637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594871199861760 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65508863 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7877514097152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70084559 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2172709731516416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9634354560512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2180587245613568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43114956 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7549 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65508857 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135593416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70084559 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2179717341950282 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7864325087274 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H:DeviceStatsMonitor.on_train_batch_end/act^HóH¬ive.all.allocated‚ 135603500 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2179715971910916 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7862980475876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135603500 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7876168653824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43115306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20118535 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20118540 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594870831770624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2180587245613568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7877514097152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70094637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594871199861760 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65508863 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7877514097152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70084559 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2172709731516416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9634354560512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2180587245613568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E·!‚(:³  ŽäŠÓÎÁ÷³ disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.821524302164715 disk.drbd1285.out‚0$ disk.nvme0n1.out‚43118.240234375 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51398905716826 disk.sda1.out‚0 disk.drbd1584.in‚0 network.recv‚890791 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd2041.out‚0$ memory_percent‚92.10772514343262 cpu‚25.326661037581488 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.sda.out‚ 355.96875 disk.sda2.out‚ 355.96875 disk.drbd1285.in‚0% disk./.usageGB‚1260.8617401123047 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1939.60546875 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125! disk.dm-0.out‚43118.240234375 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0" proc.memory.rssMB‚ 2905.2578125 network.sent‚ 1.3337936e+07 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 71.5478515625 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 93.33203125±D…4¸!/ _wandbruntime‚13783  _runtime‚13783Dª7Œ½:¹!·u ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5659996271133423 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3094591498374939 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28422296047210693 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02584996446967125 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.661019504070282 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43944603204727173 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.932421863079071 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7413546442985535 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 , Purity/0FRt1SRqqt2FBt/FBt_purity‚0.90625 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3903392255306244 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.615077555179596 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15923407673835754 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7680724859237671 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 /particle/accuracy_4_of_4‚0.9600567817687988 % _timestamp‚1.7848530205864305e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6454424858093262 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.701301634311676 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7245837450027466 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 -"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5014201998710632 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.951138576898681 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.683775782585144 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6543404459953308 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14210839569568634 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10953382402658463 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2715834975242615 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4600861668586731 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027526216581463814 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.596614420413971 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_2_of_2‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3433835506439209 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6966367959976196 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.34171441197395325 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7951359748840332 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8345959782600403 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.34165072441101074 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 epoch‚26 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.656926155090332 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7429439425468445 6"Purity/1FRt*SRqqt1FBt/event_purity‚0.5274618268013 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.34239575266838074 *jet/accuracy_2_of_4‚0.8348342776298523 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7630239129066467 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6655067205429077 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8297850489616394 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05451568588614464 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 5 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.72161865234375 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 *jet/accuracy_1_of_4‚0.9707711338996887 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17182683944702148 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9889764785766602 particle/accuracy_2_of_4‚1 _step‚630 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.649806559085846 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2715252935886383 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9165552258491516 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.381256639957428 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1684950739145279 *jet/accuracy_3_of_4‚0.1148163229227066 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21760931611061096 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2432488650083542 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6860783696174622 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958193898200989 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26122742891311646 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6837872266769409 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 (particle/accuracy‚0.8548641800880432 6validation_average_jet_accuracy‚0.4024352729320526 trainer/global_step‚7559 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7335309386253357 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 6"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7012979388237 /particle/accuracy_3_of_3‚0.9991790056228638 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9970255494117737 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28655341267585754 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8049241900444031 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1529339998960495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8038156628608704 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27600258588790894 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8352271914482117 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7471382021903992 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8718680143356323 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39180803298950195 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3425063490867615 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32129624485969543 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04468084126710892 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.376120001077652 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.862603485584259 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05320114642381668 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8697498440742493 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6124497652053833 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4608950912952423 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8006848692893982 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02966744638979435 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_1_of_2‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7132283449172974 )jet/accuracy_2_of_3‚0.602766215801239 +particle/sensitivity‚0.6403221487998962 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9377813935279846 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8031616806983948 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7597385048866272 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9910587072372437 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6417394280433655 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 +jet/accuracy_3_of_3‚0.07458456605672836 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27503320574760437 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7833694219589233 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14023831486701965 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2654416263103485 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 _runtime‚13783.94446916 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8285793662071228 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 *jet/accuracy_1_of_2‚0.7933641672134399 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9873343110084534 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49161192774772644 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2653645873069763 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.726447582244873 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6381868124008179 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.606030285358429 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0320490226149559 particle/accuracy_1_of_1‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6986886858940125 *jet/accuracy_1_of_1‚0.3673640191555023 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6528282165527344 -"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7047093510627747 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32898402214050293 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6652469635009766 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928318858146667 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21419718861579895 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.656926155090332 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25146812200546265 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33048510551452637 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8718680143356323 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7320637106895447 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.724406361579895 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07155138999223709 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 +particle/specificity‚0.9134670495986938 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33048510551452637q%X»º!¼u >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07155138999223709 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 +particle/specificity‚0.9134670495986938 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33048510551452637 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5659996271133423 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3094591498374939 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28422296047210693 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02584996446967125 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.661019504070282 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43944603204727173 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.932421863079071 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7413546442985535 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 , Purity/0FRt1SRqqt2FBt/FBt_purity‚0.90625 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3903392255306244 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.615077555179596 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15923407673835754 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7680724859237671 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 /particle/accuracy_4_of_4‚0.9600567817687988 % _timestamp‚1.7848530205864305e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6454424858093262 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.701301634311676 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7245837450027466 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 -"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5014201998710632 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.951138576898681 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.683775782585144 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6543404459953308 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14210839569568634 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10953382402658463 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2715834975242615 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4600861668586731 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027526216581463814 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.596614420413971 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_2_of_2‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3433835506439209 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6966367959976196 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.34171441197395325 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7951359748840332 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8345959782600403 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.34165072441101074 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 epoch‚26 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.656926155090332 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7429439425468445 6"Purity/1FRt*SRqqt1FBt/event_purity‚0.5274618268013 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.34239575266838074 *jet/accuracy_2_of_4‚0.8348342776298523 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7630239129066467 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6655067205429077 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8297850489616394 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05451568588614464 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 5 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.72161865234375 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 *jet/accuracy_1_of_4‚0.9707711338996887 7 0 gº" Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17182683944702148 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9889764785766602 particle/accuracy_2_of_4‚1 _step‚630 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.649806559085846 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2715252935886383 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9165552258491516 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.381256639957428 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1684950739145279 *jet/accuracy_3_of_4‚0.1148163229227066 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21760931611061096 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2432488650083542 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6860783696174622 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958193898200989 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26122742891311646 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6837872266769409 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 (particle/accuracy‚0.8548641800880432 6validation_average_jet_accuracy‚0.4024352729320526 trainer/global_step‚7559 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7335309386253357 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 6"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7012979388237 /particle/accuracy_3_of_3‚0.9991790056228638 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9970255494117737 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28655341267585754 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8049241900444031 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1529339998960495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8038156628608704 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27600258588790894 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8352271914482117 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7471382021903992 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8718680143356323 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39180803298950195 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3425063490867615 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32129624485969543 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04468084126710892 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.376120001077652 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.862603485584259 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05320114642381668 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8697498440742493 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6124497652053833 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4608950912952423 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8006848692893982 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02966744638979435 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_1_of_2‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7132283449172974 )jet/accuracy_2_of_3‚0.602766215801239 +particle/sensitivity‚0.6403221487998962 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9377813935279846 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8031616806983948 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7597385048866272 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9910587072372437 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6417394280433655 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 +jet/accuracy_3_of_3‚0.07458456605672836 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27503320574760437 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7833694219589233 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14023831486701965 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2654416263103485 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 _runtime‚13783.94446916 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8285793662071228 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 *jet/accuracy_1_of_2‚0.7933641672134399 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9873343110084534 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49161192774772644 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2653645873069763 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.726447582244873 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6381868124008179 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.606030285358429 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0320490226149559 particle/accuracy_1_of_1‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6986886858940125 *jet/accuracy_1_of_1‚0.3673640191555023 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6528282165527344 -"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7047093510627747 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32898402214050293 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6652469635009766 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928318858146667 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21419718861579895 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.656926155090332 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25146812200546265 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33048510551452637 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8718680143356323 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7320637106895447 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.724406361579895öˆQÅ»!‚(:’ äŠÓ¹Õ׳ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚297.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.17333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04<¸Z=¼!‚(:²  äŠÓ…²ô³ disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 356.515625 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 93.33203125 network.sent‚ 1.33425e+07 network.recv‚892091 disk.drbd1817.in‚0 disk.drbd1550.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0! proc.memory.rssMB‚ 2915.484375 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1929.29296875 proc.cpu.threads‚17 disk.sda1.out‚0! disk.dm-0.out‚43118.396484375 disk.drbd1356.out‚0 cpu‚25.329345915540348 disk.sda2.in‚0.03125 disk.dm-0.in‚ 93.33203125 disk.drbd2111.in‚0 disk.drbd2111.out‚0$ memory_percent‚92.14968681335449 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0% disk./.usageGB‚1260.8617401123047 disk.drbd1584.in‚0 disk.sda.out‚ 356.515625 disk.drbd1938.in‚0 disk.rbd1.out‚ 71.6103515625 disk.drbd1226.in‚0* proc.memory.percent‚11.863136291503906) disk./.usagePercent‚70.51398905716826 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0$ disk.nvme0n1.out‚43118.396484375 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚05¡­¢‘½!‚(:† ¬äŠÓŽ¥¦³ gpu.0.gpu‚85.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚295.26' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.42 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂIäA¾!‚(:¶  ¬äŠÓߌɳ network.recv‚892157 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0# disk.dm-0.out‚43118.96728515625 disk.drbd1510.in‚0 disk.sda.out‚ 356.8515625 cpu‚25.329333471437973 disk.rbd1.out‚ 71.6103515625 disk.drbd1510.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51398799047905 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0$ memory_percent‚92.15062459309897 disk.drbd1938.out‚0% disk./.usageGB‚1260.8617210388184 disk.drbd2078.out‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚43118.96728515625 disk.drbd1817.out‚0 disk.sda2.out‚ 356.8515625 disk.dm-0.in‚ 93.33203125 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0! proc.memory.rssMB‚ 2915.484375* proc.memory.percent‚11.863136291503906 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 93.33203125 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚0.03125% proc.memory.availableMB‚ 1929.0625 disk.drbd2041.out‚0 network.sent‚ 1.3342566e+07 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0"ÅZ’ž¿!‚(:“ »äŠÓù ª³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.631' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.21033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å»,ûEÀ!‚(:º  »äŠÓ—„ȳ cpu‚25.32924758072241 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0! proc.memory.rssMB‚ 2915.484375 disk.sda.out‚ 357.34765625 disk.drbd1289.in‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1550.out‚0 disk.drbd2118.in‚0& disk.nvme0n1.out‚43119.21728515625 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0# memory_percent‚92.1506404876709% disk./.usageGB‚1260.8617401123047 disk.sda1.in‚0# disk.dm-0.out‚43119.21728515625 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.nvme0n1.in‚ 93.33203125 disk.drbd1510.out‚0 proc.cpu.threads‚17 network.recv‚898443 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 357.34765625 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51398905716826 disk.drbd1584.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1929.05859375* proc.memory.percent‚11.863136291503906 network.sent‚ 1.3462067e+07 disk.rbd1.out‚ 71.9541015625 disk.drbd1285.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 +A@Á!‚(:µ  ÊäŠÓ”×Þ³ disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 72.0400390625 disk.drbd1289.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 93.33203125 cpu‚25.329179232304163* proc.memory.percent‚11.863136291503906 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.14963912963867% disk./.usageGB‚1260.8617401123047 disk.sda1.out‚0 disk.dm-0.out‚43119.55859375 disk.drbd1965.out‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.dm-0.in‚ 93.33203125 disk.drbd1584.out‚0 disk.sda.out‚ 357.69140625 disk.drbd1965.in‚0 disk.drbd2078.out‚0 network.sent‚ 1.3463188e+07 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 357.69140625 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 1929.3046875 disk.drbd1938.in‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1828.out‚0# disk.nvme0n1.out‚43119.55859375 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1285.out‚0! proc.memory.rssMB‚ 2915.484375 network.recv‚899543) disk./.usagePercent‚70.51398905716826 disk.drbd2118.out‚0õØ0žÂ!‚(:“ ÊäŠÓ³ gpu.0.gpu‚78.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.059' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.35300000000001 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iM¿ŸÃ!‚(:” ÙäŠÓþô´³ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚109.432' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.477333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p¬Wë@Ä!‚(:µ  ÙäŠÓ˜‹Õ³ disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0$ memory_percent‚92.15065638224283 disk.drbd1285.in‚0 disk.rbd1.out‚ 72.1025390625 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1584.out‚0 disk.sda2.in‚0.03125! proc.memory.rssMB‚ 2915.484375 proc.cpu.threads‚17) disk./.usagePercent‚70.51398905716826 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0( proc.memory.availableMB‚ 1929.0546875 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0# disk.nvme0n1.out‚43119.71484375 network.sent‚ 1.3465469e+07 network.recv‚899741 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 358.07421875 cpu‚25.329147930817324 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 358.07421875 disk.dm-0.in‚ 96.47265625 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8617401123047 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.dm-0.out‚43119.71484375 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.863136291503906 disk.drbd1938.out‚0¹ ^o4Å!/ _wandbruntime‚13851  _runtime‚13851À×]ÉÆ!à _runtime‚13851.916282059 _step‚631 ' lr-AdamW/pg1‚5.2420625933181605e-05 ' lr-AdamW/pg2‚5.2420625933181605e-05 trainer/global_step‚7599 $ _timestamp‚1.784853088559038e+09óO!ÒÎÇ!È ' lr-AdamW/pg2‚5.2420625933181605e-05 trainer/global_step‚7599 $ _timestamp‚1.784853088559038e+09 _runtime‚13851.916282059 _step‚631 ' lr-AdamW/pg1‚5.2420625933181605e-05÷9;Ü4È!/ _wandbruntime‚13851  _runtime‚13851‹nÞ(É!P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2188936963475456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70588800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66008278 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2188936937302016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7931954138624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43424106 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚598770931306496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9702812326912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66008278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20254792 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2188073304929200 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70588800 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2196870263050240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2196868891440640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2196868891440640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9702869885440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚598770563215360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7920017007639 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2196870263050240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20254787 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66008272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70578736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚608473375542272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7933299574784 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2195993321936839 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2188936937302016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66008278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20254792 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2188073304929200 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70588800 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2196870263050240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2196868891440640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2196868891440640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9702869885440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚598770563215360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7920017007639 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2196870263050240ž; ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20254787 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66008272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70578736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚608473375542272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7933299574784 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2195993321936839 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2188936937302016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2188936963475456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70588800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66008278 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2188936937302016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7931954138624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43424106øœ`ÃS4Ë!/ _wandbruntime‚13853  _runtime‚13853ÙBmí'Ì!ŒN J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70587818 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43429678 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2196271774473001 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7934322089984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70587818 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20257432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2189215846592512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2188352081275008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23171925 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7921037809391 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136614541 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136614541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66016645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚321 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598849149581312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7599 % _timestamp‚1.7848530900364025e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70597896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚608553622138368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2197150194855936 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2189215846592512 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2189215872765952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136604457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2197148823239168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2196273144512367 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚13853.393985076 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2188352106702976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚326 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚633 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66016639 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9704104465920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43429352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136604457 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66016645 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248ÕIÇ|'Í!‘N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598849149581312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7599 % _timestamp‚1.7848530900364025e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70597896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚608553622138368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2197150194855936 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2189215846592512 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2189215872765952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136604457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2197148823239168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2196273144512367 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚13853.393985076 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2188352106702976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚326 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚633 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66016639 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9704104465920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43429352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136604457 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66016645 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70587818 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43429678 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2196271774473001 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7934322089984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70587818 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20257432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2189215846592512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2188352081275008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23171925 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7921037809391 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136614541 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136614541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66016645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚321 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248ùÁQä4Î!/ _wandbruntime‚13853  _runtime‚13853x‡ÖÏ!‚ 1loss/SRqqt2/detection_loss‚0.5611458942196837 _step‚634 0loss/FBt2/assignment_loss‚0.0282190269831253 1loss/SRqqt1/detection_loss‚0.9021408072386096 /loss/FBt1/detection_loss‚0.4446691323408029 trainer/global_step‚7599 _runtime‚13853.430193133 /loss/FRt2/detection_loss‚1.4003474016418358 0loss/FRt2/assignment_loss‚0.6749683124435769 2loss/SRqqt1/assignment_loss‚0.4902835890390109 1loss/SRqqt2/assignment_loss‚0.566709245300381 2loss/FBt1/assignment_loss‚0.011196845749253923 /loss/FBt2/detection_loss‚0.6258646650313671 %loss/total_loss‚7.842484725967164 epoch‚27 % _timestamp‚1.7848530900729787e+09 0loss/FRt1/assignment_loss‚1.6219309013709795 /loss/FRt1/detection_loss‚0.5150089046085382.¢„Ú Ð!‡ _step‚634 0loss/FBt2/assignment_loss‚0.0282190269831253 1loss/SRqqt1/detection_loss‚0.9021408072386096 /loss/FBt1/detection_loss‚0.4446691323408029 trainer/global_step‚7599 _runtime‚13853.430193133 /loss/FRt2/detection_loss‚1.4003474016418358 0loss/FRt2/assignment_loss‚0.6749683124435769 2loss/SRqqt1/assignment_loss‚0.4902835890390109 1loss/SRqqt2/assignment_loss‚0.566709245300381 2loss/FBt1/assignment_loss‚0.011196845749253923 /loss/FBt2/detection_loss‚0.6258646650313671 %loss/total_loss‚7.842484725967164 epoch‚27 % _timestamp‚1.7848530900729787e+09 0loss/FRt1/assignment_loss‚1.6219309013709795 /loss/FRt1/detection_loss‚0.5150089046085382 1loss/SRqqt2/detection_loss‚0.5611458942196837ú¤ïÄ-4Ñ!‚(:©  èäŠÓÅ÷г disk.drbd1817.out‚0 disk.sda2.out‚ 358.453125 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0# memory_percent‚92.1498934427897) disk./.usagePercent‚70.51398841715474 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0( proc.memory.availableMB‚ 1929.2421875 network.sent‚ 1.3471248e+07* proc.memory.percent‚11.863136291503906 network.recv‚901753 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1285.out‚0! proc.memory.rssMB‚ 2915.484375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.dm-0.out‚ 43120.765625 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0$ disk./.usageGB‚1260.861728668213 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1552.out‚0 disk.sda.out‚ 358.453125 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 72.1025390625 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1965.out‚0! disk.nvme0n1.out‚ 43120.765625 disk.drbd2118.in‚0 cpu‚25.3290986067356 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0ã{=“Ò!‚(:ˆ èäŠÓ¤®ª³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.857' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.619 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—¸¬ìžÓ!‚(:“ ÷äŠÓ²ø´³ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.918' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.97266666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Î‘×Ô!‚(:©  ÷äŠÓ¸ù׳) disk./.usagePercent‚70.51399055053315 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.dm-0.out‚43121.10546875 disk.drbd1226.out‚0 network.recv‚903511 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1928.1796875 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 72.322265625 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.nvme0n1.in‚ 96.47265625 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0$ memory_percent‚92.15421676635742! proc.memory.rssMB‚ 2915.484375 disk.drbd1828.in‚0 disk.drbd1552.in‚0% disk./.usageGB‚1260.8617668151855 disk.drbd1938.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.out‚358.875 disk.sda2.in‚0.03125 disk.drbd2118.out‚0* proc.memory.percent‚1_îÍ1.863136291503906 proc.cpu.threads‚17 network.sent‚ 1.3528427e+07 disk.drbd1285.out‚0# disk.nvme0n1.out‚43121.10546875 cpu‚25.32901311370963 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚358.875 disk.sda1.in‚0%\zªŸÕ!‚(:” †åŠÓõï׳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚309.856' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.28533333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªˆÈBÖ!‚(:·  †åŠÓ»×ú³ disk.drbd1510.out‚0 disk.sda2.out‚ 359.36328125 disk.drbd1289.out‚0* proc.memory.percent‚11.863136291503906 network.recv‚905492 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125( proc.memory.availableMB‚ 1928.3203125 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.sda.out‚ 359.36328125 disk.drbd2078.out‚0! proc.memory.rssMB‚ 2915.484375 disk.drbd2118.in‚0) disk./.usagePercent‚70.51381049339547 disk.drbd1828.out‚0 disk.rbd1.out‚ 72.556640625% disk.nvme0n1.out‚43121.4970703125 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.sda.in‚0.03125 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0" disk.dm-0.out‚43121.4970703125 disk.drbd2111.in‚0$ memory_percent‚92.15364456176758 cpu‚25.32894488253092 network.sent‚ 1.3534171e+07 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 96.47265625 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8585472106934 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚00üO8ž×!‚(:“ •åŠÓϚ³ gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.042' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.34733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬¥Ž'AØ!‚(:¶  •åŠÓʙ㳠disk.sda.in‚0.03125 disk.sda2.in‚0.03125" disk.dm-0.out‚43121.6494140625 disk.drbd1285.in‚0$ memory_percent‚92.15553601582846! proc.memory.rssMB‚ 2915.484375 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚43121.6494140625 disk.drbd1673.in‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51380814667922 disk.drbd1550.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2078.out‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.863136291503906 disk.drbd1510.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8585052490234 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1927.85546875 cpu‚25.32889649868383 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 359.7421875 disk.sda2.out‚ 359.7421875 disk.rbd1.out‚ 72.634765625 proc.cpu.threads‚17 network.recv‚905624 disk.drbd1226.in‚0 network.sent‚ 1.3534977e+07 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 96.47265625œ~¿ûÙ!‚(:’ ¤åŠÓøö­³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚304.582' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.52733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë`VCÚ!‚(:¸  ¤åŠÓÐòг disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1673.in‚0$ memory_percent‚92.15556780497232 disk.drbd1550.out‚0 disk.sda.out‚ 360.1640625 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 360.1640625$ disk./.usageGB‚1260.856777191162 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2078.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51371150463731 disk.drbd2041.out‚0 network.sent‚ 1.3535085e+07 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0! proc.memory.rssMB‚ 2915.484375 cpu‚25.328830136999187# disk.dm-0.out‚43122.01611328125 disk.drbd1828.out‚0& disk.nvme0n1.out‚43122.01611328125 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 network.recv‚905732 disk.drbd1817.out‚0 disk.drbd1931.in‚0) proc.memory.availableMB‚ 1927.84765625 disk.drbd1226.in‚0* proc.memory.percent‚11.863136291503906 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.rbd1.out‚ 72.634765625 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1965.in‚0Ѹm“Û!‚(:ˆ ³åŠÓ»éÁ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.261' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.087 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êvÔ°@Ü!‚(:µ  ³åŠÓʹâ³ disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 360.421875 disk.drbd1510.out‚0 disk.drbd1584.out‚0 network.sent‚ 1.354332e+07 disk.drbd1550.out‚0! proc.memory.rssMB‚ 2915.484375$ memory_percent‚92.15561548868814 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚43122.63232421875 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1289.in‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.863136291503906% disk./.usageGB‚1260.8530311584473 disk.drbd2041.out‚0 disk.sda.out‚ 360.421875 disk.drbd2078.out‚0 disk.rbd1.out‚ 72.7685546875 disk.sda1.out‚0( proc.memory.availableMB‚ 1927.8359375 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 cpu‚25.328744974897514 network.recv‚908905 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.sda.in‚0.03125( disk./.usagePercent‚70.5135020068776 disk.drbd1226.out‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚43122.63232421875 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚06mà4Ý!/ _wandbruntime‚13938  _runtime‚13938!%uÈÞ! & lr-AdamW/pg2‚5.175424071012201e-05 trainer/global_step‚7649 % _timestamp‚1.7848531754992452e+09 _runtime‚13938.856482216 _step‚635 & lr-AdamW/pg1‚5.175424071012201e-05 qgÈÍß!Ç _runtime‚13938.856482216 _step‚635 & lr-AdamW/pg1‚5.175424071012201e-05 & lr-AdamW/pg2‚5.175424071012201e-05 trainer/global_step‚7649 % _timestamp‚1.7848531754992452e+09û»Ým4à!/ _wandbruntime‚13938  _runtime‚13938)±*©(á!‚P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137461310 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137471380 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71034688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23317166 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2202013393618000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7971120890447 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2210865545521664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7983143694848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602699881512960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66426622 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9765318551040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71044752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2202882401826816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚338 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2209984514508447 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7969776279161 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848531755003226e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20386792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2210866917131264 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2202882401826816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7983143694848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71034688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2210866917131264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9765367720960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20386787 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623÷= (â!‡P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71034688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23317166 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2202013393618000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7971120890447 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2210865545521664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7983143694848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602699881512960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66426622 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9765318551040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71044752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2202882401826816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚338 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2209984514508447 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7969776279161 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848531755003226e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20386792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2210866917131264 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2202882401826816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7983143694848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71034688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2210866917131264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9765367720960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20386787 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137461310 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137471380 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0üŒæüé4ã!/ _wandbruntime‚13940  _runtime‚13940À€Î=µ ä!ŒN RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2211145477320192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20389432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9766614877184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66434995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚13940.332793153 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7984166202880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚612545450847232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7985511646208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23320073 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚332 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7984166202880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚612545031496192 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2210262967044609 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137478759 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2211146848936960 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2211146848936960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137488843 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137488843 trainer/global_step‚7649 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71043770 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43709505 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602778467878912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43709168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22022921953917761÷ú÷'å!‘N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2203161311117312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66434989 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71053848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7970797080801 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66434995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7972141692199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71043770 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43709168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2202292195391776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2211145477320192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20389432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9766614877184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66434995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚13940.332793153 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7984166202880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚612545450847232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7985511646208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23320073 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚332 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7984166202880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚612545031496192 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2210262967044609 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137478759 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2211146848936960 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2211146848936960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137488843 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137488843 trainer/global_step‚7649 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71043770 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43709505 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602778467878912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 Qé!‚(:³  ÂåŠÓ×µî³ disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0$ disk./.usageGB‚1260.853313446045 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0# disk.nvme0n1.out‚43123.10546875 disk.drbd1289.out‚0) disk./.usagePercent‚70.51351779387782 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0$ memory_percent‚92.15497970581055 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1828.out‚0 cpu‚25.32871378343337 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 72.9482421875 network.recv‚909894 disk.dm-0.out‚43123.10546875 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0( proc.memory.availableMB‚ 1927.9921875 network.sent‚ 1.3544289e+07! proc.memory.rssMB‚ 2915.484375 disk.drbd2118.out‚0 disk.sda.out‚ 360.73828125 disk.sda2.out‚ 360.73828125 disk.drbd2111.out‚0* proc.memory.percent‚11.863136291503906_¦—žê!‚(:“ ÂåŠÓó²Í³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.008' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.00266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß!nýŸë!‚(:” ÑåŠÓÜê°³ gpu.0.gpu‚58.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.371' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.79033333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Mˆè=ì!‚(:²  ÑåŠÓ´…Ó³ disk.dm-0.in‚ 96.47265625 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8532943725586 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.863136291503906 disk.sda2.out‚ 361.0078125 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 73.0107421875! proc.memory.rssMB‚ 2915.484375) disk./.usagePercent‚70.51351672718862 disk.drbd1356.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.sda.out‚ 361.0078125( proc.memory.availableMB‚ 1927.7109375 cpu‚25.32866596107073 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.dm-0.out‚43123.26953125 network.recv‚911621 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1931.out‚0 network.sent‚ 1.3603527e+07 disk.drbd1552.out‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0# disk.nvme0n1.out‚43123.26953125 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1226.out‚0$ memory_percent‚92.15612411499023úõ’Êží!‚(:“ àåŠÓˆ‡¿³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.643' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.21433333333331 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÑ+Š?î!‚(:´  àåŠÓ‚Œß³ disk.drbd1584.in‚0 cpu‚25.32861708161507 disk.drbd1817.out‚0% disk./.usageGB‚1260.8532752990723 disk.dm-0.in‚ 96.47265625 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd2118.out‚0 disk.sda.out‚ 361.421875 network.sent‚ 1.3603593e+07 proc.cpu.threads‚17 network.recv‚911687 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0) disk./.usagePercent‚70.51351566049942" disk.dm-0.out‚43123.6591796875 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚43123.6591796875 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.sda.in‚0.03125! proc.memory.rssMB‚ 2915.484375 disk.drbd1938.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.15612411499023* proc.memory.percent‚11.863136291503906 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 73.0107421875 disk.sda2.out‚ 361.421875 disk.drbd1965.out‚0 disk.drbd1550.in‚0( proc.memory.availableMB‚ 1927.7109375$=e¶?ï!‚(:´  ïåŠÓ¯Êͳ disk.drbd1828.out‚0! proc.memory.rssMB‚ 2915.484375 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125" disk.dm-0.out‚43123.8583984375 disk.drbd1356.in‚0' proc.memory.availableMB‚ 1928.890625 disk.nvme0n1.in‚ 96.47265625 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 cpu‚25.32860483163951 disk.sda1.out‚0% disk./.usageGB‚1260.8533172607422% disk.nvme0n1.out‚43123.8583984375 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 361.64453125 network.recv‚915072 disk.drbd1817.in‚0 disk.sda.out‚ 361.64453125 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0) disk./.usagePercent‚70.51351800721567 disk.drbd1226.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.15132395426431 disk.sda1.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.863136291503906 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1828.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 73.14453125 disk.drbd1550.in‚0 disk.drbd1938.in‚0 network.sent‚ 1.361246e+07 disk.drbd1550.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17„ ´“ð!‚(:ˆ ïåŠÓÈÉ«³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚291.792' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.264 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dDlžñ!‚(:“ þåŠÓîþ³³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.979' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.32633333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p+Dò!‚(:¹  þåŠÓä¯Õ³ disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 73.14453125 cpu‚25.32853844174394 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0$ memory_percent‚92.15232531229654! proc.memory.rssMB‚ 2915.484375 disk.drbd2118.out‚0& disk.nvme0n1.out‚43124.23779296875 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 network.sent‚ 1.3614778e+07 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 361.98828125 network.recv‚915312% disk./.usageGB‚1260.8532981872559# disk.dm-0.out‚43124.23779296875 disk.drbd2078.in‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1928.64453125* proc.memory.percent‚11.863136291503906 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 361.98828125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51351694052647 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 96.47265625v!+ÁAó!‚(:¶  æŠÓ±Ç˳% disk./.usageGB‚1260.8534889221191 disk.drbd1510.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.15232531229654 disk.dm-0.in‚ 96.47265625 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 73.20703125 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.863136291503906 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1928.64453125 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 network.sent‚ 1.3614844e+07 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd1285.out‚0 disk.sda.out‚ 362.41796875 disk.drbd1931.out‚0 disk.sda2.out‚ 362.41796875 disk.drbd1289.out‚0 cpu‚25.32845492912529 proc.cpu.threads‚17 network.recv‚915378( disk./.usagePercent‚70.5135276074185 disk.rbd1.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1584.in‚0 disk.drbd1817.in‚0! proc.memory.rssMB‚ 2915.484375" disk.dm-0.out‚43124.4404296875 disk.drbd1828.in‚0 disk.drbd1550.in‚0% disk.nvme0n1.out‚43124.4404296875 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd2041.out‚0‹C„>žô!‚(:“ æŠÓÌ·–´ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.549' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.84966666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á×+Ì4õ!/ _wandbruntime‚14025  _runtime‚14025Ï_·ÂÇö!Á $ _timestamp‚1.784853262098794e+09 _runtime‚14025.456015639 _step‚639 & lr-AdamW/pg1‚5.108874815753447e-05 & lr-AdamW/pg2‚5.108874815753447e-05 trainer/global_step‚7699óó!Ì÷!Æ & lr-AdamW/pg1‚5.108874815753447e-05 & lr-AdamW/pg2‚5.108874815753447e-05 trainer/global_step‚7699 $ _timestamp‚1.784853262098794e+09 _runtime‚14025.456015639 _step‚639ÿAeN4ø!/ _wandbruntime‚14025  _runtime‚14025´±+r„ù!‚P WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8034333251072 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43982143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8022224773255 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2224862199602688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616457189158912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616456767703552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8035678687232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138335612 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2215953482306800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66844978 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8034333251072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚14025.457307874 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2216827892525056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2224862199602688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9827621257216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43982143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8022224773255 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2224862199602688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616457189158912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616456767703552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8035678687232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138335612 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2215953482306800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66844978 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8034333251072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0€ì¤k4û!/ _wandbruntime‚14026  _runtime‚14026³ø02'ü!ŒN H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23466609 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66853339 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138363145 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71499722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20521427 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚616537021305856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71509800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 % _timestamp‚1.7848532635770972e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138353061 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2216232284080576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2224255529655583 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8021900963609 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9828867038208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71509800 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2217106801815552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚606707786176512 _runtime‚14026.934698139 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66853345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚606708154267648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43987688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20521432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9828822069760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66853339 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2217106801815552³˜¬ý!‘N EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138353061 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2216232284080576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2224255529655583 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8021900963609 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9828867038208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71509800 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2217106801815552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚606707786176512 _runtime‚14026.934698139 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66853345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚606708154267648 Lÿ!h&DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43987688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20521432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9828822069760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66853339 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2217106801815552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23466609 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66853339 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138363145 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71499722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20521427 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚616537021305856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71509800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 % _timestamp‚1.7848532635770972e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0±•LL4þ!/ _wandbruntime‚14026  _runtime‚14026û.ê¨ ÿ!„ 2loss/SRqqt2/assignment_loss‚0.4408530472109772 1loss/SRqqt1/detection_loss‚0.8944574834321899 1loss/SRqqt2/detection_loss‚0.6008301657690603 _runtime‚14026.970693384 _step‚642 2loss/SRqqt1/assignment_loss‚0.4404972696381863 %loss/total_loss‚7.975365768525453 trainer/global_step‚7699 epoch‚27 2loss/FBt1/assignment_loss‚0.009850011587294224 1loss/FBt2/assignment_loss‚0.01698589307088118 /loss/FRt1/detection_loss‚0.5197434259793317 /loss/FRt2/detection_loss‚1.3646493973646694 /loss/FBt2/detection_loss‚0.5124009010171524 % _timestamp‚1.7848532636134825e+09 0loss/FRt1/assignment_loss‚1.7321990231969464 0loss/FRt2/assignment_loss‚0.7813703854099597 /loss/FBt1/detection_loss‚0.6615287648488043w°Ì€"‰ 2loss/FBt1/assignment_loss‚0.009850011587294224 1loss/FBt2/assignment_loss‚0.01698589307088118 /loss/FRt1/detection_loss‚0.5197434259793317 /loss/FRt2/detection_loss‚1.3646493973646694 /loss/FBt2/detection_loss‚0.5124009010171524 % _timestamp‚1.7848532636134825e+09 0loss/FRt1/assignment_loss‚1.7321990231969464 0loss/FRt2/assignment_loss‚0.7813703854099597 /loss/FBt1/detection_loss‚0.6615287648488043 2loss/SRqqt2/assignment_loss‚0.4408530472109772 1loss/SRqqt1/detection_loss‚0.8944574834321899 1loss/SRqqt2/detection_loss‚0.6008301657690603 _runtime‚14026.970693384 _step‚642 2loss/SRqqt1/assignment_loss‚0.4404972696381863 %loss/total_loss‚7.975365768525453 trainer/global_step‚7699 epoch‚27‚­þÅFŸ"‚(:” œæŠÓï­Ä³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.329' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.77633333333334 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î+l±;‚"‚(:°  œæŠÓº†ã³ disk.drbd1938.in‚0) disk./.usagePercent‚70.51353827431055 disk.drbd1552.in‚0 disk.rbd1.out‚ 73.20703125 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.sda2.out‚362.9375 disk.drbd2078.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚43126.18603515625 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚43126.18603515625 disk.drbd1356.out‚0! proc.memory.rssMB‚ 2915.484375 disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1965.in‚0% disk./.usageGB‚1260.8536796569824 disk.drbd1289.out‚0 disk.drbd1550.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1928.58203125 disk.sda1.in‚0 disk.sda.out‚362.9375* proc.memory.percent‚11.863136291503906 network.sent‚ 1.3615286e+07 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 96.47265625 network.recv‚916306 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0# memory_percent‚92.1525796254476 disk.drbd1817.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.rbd1.in‚0 cpu‚25.32840614356304Õè릞ƒ"‚(:“ «æŠÓ¶Ê¿³ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.257' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.08566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž¢0»D„"‚(:¹  «æŠÓù´Ý³ disk.drbd1938.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1510.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0! proc.memory.rssMB‚ 2915.484375 disk.sda.out‚ 363.19921875 disk.sda2.out‚ 363.19921875 disk.drbd2078.in‚0 disk.drbd1226.in‚0 network.recv‚920092 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.863136291503906 disk.dm-0.in‚ 96.47265625# disk.dm-0.out‚43126.51416015625 disk.drbd1285.in‚0( proc.memory.availableMB‚ 1928.3359375 disk.drbd1550.in‚0$ memory_percent‚92.15358098347983 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 network.sent‚ 1.3681376e+07% disk./.usageGB‚1260.8539428710938 disk.drbd2078.out‚0 disk.rbd1.out‚ 73.46484375 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0& disk.nvme0n1.out‚43126.51416015625 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51355299462156 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 cpu‚25.328322833268857 disk.sda1.in‚00Q::…"‚(:¯  ºæŠÓþ disk.drbd1285.in‚0# disk./.usageGB‚1260.85396194458 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 network.recv‚921070 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 363.59375 disk.drbd2111.in‚0) disk./.usagePercent‚70.51355406131077 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 73.55078125! proc.memory.rssMB‚ 2915.484375 disk.drbd1510.in‚0# disk.dm-0.out‚43128.02197265625 disk.drbd2041.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚43128.02197265625 disk.sda2.out‚ 363.59375 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1817.in‚0 network.sent‚ 1.3682418e+07 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0$ memory_percent‚92.15157826741537 cpu‚25.32827395400489 disk.drbd1931.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1285.out‚0' proc.memory.availableMB‚ 1928.828125 disk.drbd1931.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.863136291503906þVb“”†"‚(:‰ ºæŠÓœÒܳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚300.996' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+è‹ø@‡"‚(:µ  ÉæŠÓéÝÙ³ disk.nvme0n1.in‚ 96.47265625 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.dm-0.in‚ 96.47265625 network.recv‚921136 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚43128.24462890625 disk.sda1.in‚0 disk.drbd1510.out‚0 cpu‚25.328209635817117 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 73.79296875 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1550.in‚0# memory_percent‚92.1525796254476) proc.memory.availableMB‚ 1928.58203125 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 364.078125 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.863136291503906% disk./.usageGB‚1260.8538665771484 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 364.078125# disk.dm-0.out‚43128.24462890625 network.sent‚ 1.3682484e+07 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1828.out‚0! proc.memory.rssMB‚ 2915.484375 proc.cpu.threads‚17) disk./.usagePercent‚70.51354872786474 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125-‹çÛŸˆ"‚(:” ÉæŠÓ©Ò¾³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.532' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.17733333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zš´ž‰"‚(:“ ØæŠÓôܵ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚66.0 gpu.0.powerWatts‚291.189' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.06300000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ur ’/Š"‚(:¤  ØæŠÓÂêÕ³ disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 364.55078125 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1673.in‚0 disk.drbd1289.in‚0 cpu‚25.328144438485452 network.recv‚922139) disk./.usagePercent‚70.51354766117555 disk.drbd1938.in‚0 disk.sda.in‚0.03125) proc.memory.availableMB‚ 1928.58203125 disk.rbd1.in‚0 disk.rbd1.out‚ 73.79296875 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.sda.out‚ 364.55078125$ disk./.usageGB‚1260.853847503662 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.863136291503906 disk.drbd2041.in‚0 disk.sda2.in‚0.03125 proc.cpu.threads‚17 network.sent‚ 1.3684974e+07 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0! proc.memory.rssMB‚ 2915.484375 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 96.47265625 disk.dm-0.out‚43128.5 disk.drbd1828.out‚0 disk.nvme0n1.out‚43128.5 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0# memory_percent‚92.1525796254476 disk.drbd1938.out‚0 disk.drbd1965.in‚03õ»4‹"/ _wandbruntime‚14112  _runtime‚14112”¥+æÈŒ" _step‚643 & lr-AdamW/pg1‚5.042423550624549e-05 & lr-AdamW/pg2‚5.042423550624549e-05 trainer/global_step‚7749 % _timestamp‚1.7848533488663137e+09 _runtime‚14112.224899144B‹#>Í"Ç & lr-AdamW/pg1‚5.042423550624549e-05 & lr-AdamW/pg2‚5.042423550624549e-05 trainer/global_step‚7749 % _timestamp‚1.7848533488663137e+09 _runtime‚14112.224899144 _step‚643ƒÀÒï4Ž"/ _wandbruntime‚14112  _runtime‚14112šF½("‚P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23610498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67263328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8086868243456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2230773330876416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8071984044777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20650787 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8085522807296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139209914 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71956656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9889888821760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620448345177088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71956656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44260908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8073328656063 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2237965529612409 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 % _timestamp‚1.7848533488670745e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2230773330876416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44261290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2238858853683712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610558518108160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23610121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610558886199296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9889827068928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8086868243456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20650792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0qÜØÙÛ"‡P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67263328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8086868243456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2230773330876416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8071984044777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20650787 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8085522807296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139209914 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71956656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9889888821760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620448345177088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71956656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44260908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8073328656063 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2237965529612409 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 % _timestamp‚1.7848533488670745e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2230773330876416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split..(CÞ2all.allocated‚44261290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2238858853683712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610558518108160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23610121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610558886199296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9889827068928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8086868243456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20650792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23610498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832„Ôã’î4‘"/ _wandbruntime‚14113  _runtime‚14113N8*'’"‹N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610637104474112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891069235200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67271695 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2239140157099008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚7749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71955674 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚14113.699662289 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2230172347341408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784853350342072e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2231052240166912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139237447 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8074349457815 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44266857 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139227363 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67271689 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2231052266340352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620528597254656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620528173709312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2230172372769376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8073004846417 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8087890758656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891124689408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8087890758656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139227363 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚367 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2239140157099008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8086545315328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8086545315328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44266485 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23613058 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238246722227191 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20653427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2239138785482240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71965752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71955674 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623&Ø‘'“"N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20653427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2239138785482240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71965752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71955674 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610637104474112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891069235200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67271695 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2239140157099008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚7749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71955674 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚14113.699662289 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2230172347341408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784853350342072e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2231052240166912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139237447 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8074349457815 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44266857 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139227363 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67271689 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2231052266340352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620528597254656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620528173709312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2230172372769376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8073004846417 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8087890758656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891124689408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8087890758656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139227363 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚367 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2239140157099008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8086545315328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8086545315328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44266485 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23613058 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238246722227191 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864…‰Òµº4”"/ _wandbruntime‚14113  _runtime‚14113¿Ý^r•"ÿ 0loss/FRt1/assignment_loss‚1.6432384515930893 -loss/FBt1/detection_loss‚0.74616905163774 %loss/total_loss‚8.045243238309205 $ _timestamp‚1.784853350379751e+09 _runtime‚14113.736906645 0loss/FRt2/assignment_loss‚0.6865909255512769 2loss/SRqqt2/assignment_loss‚0.4780507292071647 1loss/FBt1/assignment_loss‚0.06294817992377132 /loss/FBt2/detection_loss‚0.4361965401000194 /loss/FRt1/detection_loss‚0.5333323869270493 .loss/FRt2/detection_loss‚1.413758309233551 1loss/SRqqt1/detection_loss‚0.9392800412643371 0loss/SRqqt2/detection_loss‚0.610337361571212 3loss/SRqqt1/assignment_loss‚0.48482796986695575 1loss/FBt2/assignment_loss‚0.01051329143303758 epoch‚27 trainer/global_step‚7749 _step‚646]­û© –"„ /loss/FRt1/detection_loss‚0.5333323869270493 .loss/FRt2/detection_loss‚1.413758309233551 1loss/SRqqt1/detection_loss‚0.9392800412643371 0loss/SRqqt2/detection_loss‚0.610337361571212 3loss/SRqqt1/assignment_loss‚0.48482796986695575 1loss/FBt2/assignment_loss‚0.01051329143303758 epoch‚27 trainer/global_step‚7749 _step‚646 0loss/FRt1/assignment_loss‚1.6432384515930893 -loss/FBt1/detection_loss‚0.74616905163774 %loss/total_loss‚8.045243238309205 $ _timestamp‚1.784853350379751e+09 _runtime‚14113.736906645 0loss/FRt2/assignment_loss‚0.6865909255512769 2loss/SRqqt2/assignment_loss‚0.4780507292071647 1loss/FBt1/assignment_loss‚0.06294817992377132 /loss/FBt2/detection_loss‚0.4361965401000194†¸s785—"‚(:ª  çæŠÓÝŒù³ disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1226.in‚0! proc.memory.rssMB‚ 2915.484375 disk.drbd2118.out‚0 disk.sda.out‚ 364.8046875 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.nvme0n1.out‚ 43128.6875 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 1929.1640625 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 73.9267578125 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1828.out‚0 network.sent‚ 1.368504e+07* proc.memory.percent‚11.863136291503906 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51354872786474% disk./.usageGB‚1260.8538665771484 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.dm-0.out‚ 43128.6875 disk.drbd2111.in‚0$ memory_percent‚92.15021133422852 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 cpu‚25.328095895896073 proc.cpu.threads‚17 network.recv‚922205 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 364.8046875 disk.rbd1.in‚0ÑÕ¶“˜"‚(:ˆ çæŠÓä׳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.397' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.799 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ᘽºŸ™"‚(:” öæŠÓ€ç±³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚308.102' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.70066666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌDŽtAš"‚(:¶  öæŠÓáѳ disk.drbd2111.in‚0! proc.memory.rssMB‚ 2915.484375 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8538665771484 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 network.sent‚ 1.3690607e+07 disk.nvme0n1.in‚ 96.47265625 disk.drbd1226.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1938.in‚0$ memory_percent‚92.15019543965658 disk.sda1.out‚0 disk.sda.out‚ 365.15625 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1929.16796875 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0# disk.dm-0.out‚43129.17041015625 disk.drbd2078.in‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 cpu‚25.328049643477456 disk.drbd1285.in‚0& disk.nvme0n1.out‚43129.17041015625 disk.drbd1931.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0* proc.memory.percent‚11.863136291503906 disk.drbd2118.in‚0 disk.sda2.out‚ 365.15625 network.recv‚923285 disk.rbd1.out‚ 73.9267578125 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51354872786474 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0¼Œ.¬œ›"‚(:‘ …çŠÓ߸®³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚274.2' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.39999999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çfz¾‘œ"‚(:®  …çŠÓ¸ÿͳ disk.drbd2078.out‚0 disk.drbd2111.out‚0) disk./.usagePercent‚70.51354872786474 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.863136291503906& disk.nvme0n1.out‚43129.35791015625 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0HX¨ disk.drbd1289.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1510.out‚0 disk.sda2.out‚365.5$ memory_percent‚92.15019543965658# disk.dm-0.out‚43129.35791015625 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0) proc.memory.availableMB‚ 1929.16796875% disk./.usageGB‚1260.8538665771484 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.in‚0.03125! proc.memory.rssMB‚ 2915.484375 disk.drbd1285.out‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 73.9892578125 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.sda.out‚365.5 disk.drbd1673.out‚0 network.recv‚926107 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 cpu‚25.328002449896093 network.sent‚ 1.3753149e+07 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 96.47265625^œŽãA"‚(:¶  ”çŠÓÉÚÕ³ disk.drbd2111.in‚0 disk.drbd1226.in‚0 network.sent‚ 1.3753215e+07 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.15032259623209) disk./.usagePercent‚70.51354787451338 disk.drbd1552.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 network.recv‚926173 disk.drbd2118.in‚0) proc.memory.availableMB‚ 1929.13671875 proc.cpu.threads‚17 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1828.in‚0% disk./.usageGB‚1260.8538513183594 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0$ disk.nvme0n1.out‚43129.650390625! proc.memory.rssMB‚ 2915.484375* proc.memory.percent‚11.863136291503906 disk.sda1.out‚0 disk.sda2.out‚ 365.7421875 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0! disk.dm-0.out‚43129.650390625 cpu‚25.327955010358643 disk.sda.out‚ 365.7421875 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1828.out‚0 disk.rbd1.out‚ 73.9892578125 disk.drbd1356.in‚0£ï?Û“ž"‚(:ˆ ”çŠÓ¨ï°³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚285.528' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 §µAŸ"‚(:¶  £çŠÓ‚‰à³ disk.drbd2118.out‚0 disk.drbd1828.in‚0$ disk./.usageGB‚1260.853931427002 disk.drbd1584.out‚0 disk.sda2.out‚ 366.19140625 disk.drbd1965.out‚0* proc.memory.percent‚11.863136291503906 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1673.in‚0! disk.dm-0.out‚43129.841796875 disk.drbd1965.in‚0 disk.drbd2111.out‚0 network.sent‚ 1.3761772e+07 disk.drbd1289.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51355235460804 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd1285.out‚0$ disk.nvme0n1.out‚43129.841796875 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2078.out‚0 disk.rbd1.out‚ 74.123046875 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0$ memory_percent‚92.14933713277182 cpu‚25.327890006558164 network.recv‚928247 disk.sda.out‚ 366.19140625 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1929.37890625! proc.memory.rssMB‚ 2915.484375 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0MOŸ "‚(:” £çŠÓÒ¾³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚302.113' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.70433333333332 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ŽÉמ¡"‚(:“ ²çŠÓ“°¨³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.831' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.94366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•¬³C¢"‚(:¸  ²çŠÓÇÜʳ# disk.dm-0.out‚43130.14013671875 disk.nvme0n1.in‚ 96.47265625& disk.nvme0n1.out‚43130.14013671875 network.recv‚928313 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0$ memory_percent‚92.15033849080405 disk.drbd2111.out‚0 cpu‚25.3278084876857 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51355128791884 disk.sda.out‚ 366.57421875 disk.drbd1673.in‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 network.sent‚ 1.3761838e+07 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0( proc.memory.availableMB‚ 1929.1328125 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 366.57421875 disk.drbd2078.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0! proc.memory.rssMB‚ 2915.484375% disk./.usageGB‚1260.8539123535156 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 74.123046875* proc.memory.percent‚11.863136291503906«þ&'4£"/ _wandbruntime‚14198  _runtime‚14198Ðâúfɤ"à ' lr-AdamW/pg1‚4.9760789858638894e-05 ' lr-AdamW/pg2‚4.9760789858638894e-05 trainer/global_step‚7799 $ _timestamp‚1.784853435425452e+09 _runtime‚14198.782708726 _step‚647sSé Î¥"È trainer/global_step‚7799 $ _timestamp‚1.784853435425452e+09 _runtime‚14198.782708726 _step‚647 ' lr-AdamW/pg1‚4.9760789858638894e-05 ' lr-AdamW/pg2‚4.9760789858638894e-05‡öÏs4¦"/ _wandbruntime‚14198  _runtime‚141985Â(§"‚P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚614488204496896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2252855507764736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72412608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140084216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2244718795401216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9952089512960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2244718795401216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614487836405760(à ` (¨"‡P PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67681678 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2244718821574656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72402544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23757292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚614488204496896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2252855507764736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72412608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140084216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2244718795401216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9952089512960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2244718795401216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614487836405760 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72421704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23760193 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2253136811180032 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67690039 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44545239 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44545625 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8139080314880 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2244997730865152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614566790862848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚614566422771712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624520124443648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624519692509696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140101665 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67690045 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72421704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67690045 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9953333580800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2252236544759433 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚¸p— 67690039 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚14200.455273382 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2244997704691712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20785427 % _timestamp‚1.7848534370976844e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140111749 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9953269737984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8125453340623 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8139080314880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2252237914798799 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140111749 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2253136811180032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140101665 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8137734871552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136osÝ'«"‘N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23759812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2244112461458176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2244997704691712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚386 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8124108729225 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2244997730865152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140111749 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9953269737984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8125453340623 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8139080314880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2252237914798799 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140111749 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2253136811180032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140101665 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8137734871552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2244112436030208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72421704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23760193 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2253136811180032 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67690039 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44545239 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44545625 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8139080314880 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2244997730865152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614566790862848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚614566422771712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624520124443648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624519692509696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140101665 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67690045 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72421704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67690045 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9953333580800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2252236544759433 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67690039 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚14200.455273382 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2244997704691712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20785427 % _timestamp‚1.7848534370976844e+09 EÅ6ð®"‹ trainer/global_step‚7799 _step‚650 0loss/FBt2/detection_loss‚0.35077532134658085 %loss/total_loss‚7.697516739500658 epoch‚27 0loss/FRt1/assignment_loss‚1.6244281297956065 2loss/SRqqt2/assignment_loss‚0.4823785500035175 1loss/SRqqt1/detection_loss‚0.8253951152437367 3loss/SRqqt1/assignment_loss‚0.46304813027098946 /loss/FRt1/detection_loss‚0.5198911291364385 /loss/FBt1/detection_loss‚0.6993823227557269 % _timestamp‚1.7848534371373234e+09 _runtime‚14200.494539784 0loss/FRt2/assignment_loss‚0.7423767535002915 2loss/FBt1/assignment_loss‚0.020158535850304537 2loss/FBt2/assignment_loss‚0.010583261428420596 /loss/FRt2/detection_loss‚1.3792743695951946 0loss/SRqqt2/detection_loss‚0.579825120573851ŠªÆ{LA¯"‚(:¶  ÁçŠÓÙ労# disk.dm-0.out‚43130.33544921875 disk.drbd1817.out‚0 disk.drbd1931.out‚0! proc.memory.rssMB‚ 2915.484375 network.sent‚ 1.3764287e+07 disk.sda.out‚ 367.0390625 disk.drbd1673.in‚0( proc.memory.availableMB‚ 1929.0859375 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 367.0390625 disk.sda2.in‚0.03125$ disk./.usageGB‚1260.853874206543 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.15052922566733 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0& disk.nvme0n1.out‚43130.33544921875 network.recv‚929212 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.863136291503906 disk.nvme0n1.in‚ 96.47265625 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 74.185546875 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51354915454043 cpu‚25.32779488273608KÐ¥Fž°"‚(:“ ÁçŠÓ°øì³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.622' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.20733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ڜƞ±"‚(:“ ÐçŠÓ˜Îɳ gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.123' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.70766666666665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f!ÈA²"‚(:¶  ÐçŠÓæ£ÿ³ disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 367.3515625 disk.drbd2078.in‚0 disk.drbd1226.out‚0 network.recv‚930441 disk.drbd1356.out‚0 disk.sda.in‚0.03125 disk.drbd1965.out‚0! proc.memory.rssMB‚ 2915.484375 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51358670200042 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.8545455932617 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 cpu‚25.327783547077306 disk.sda2.in‚0.03125) proc.memory.availableMB‚ 1928.81640625 network.sent‚ 1.3767927e+07 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 74.185546875 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0% disk.nvme0n1.out‚43131.1708984375 disk.drbd1552.out‚0" disk.dm-0.out‚43131.1708984375 disk.sda1.in‚0 disk.drbd1673.out‚0# memory_percent‚92.1516259511312 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.sda2.out‚ 367.3515625* proc.memory.percent‚11.863136291503906y(ºó”³"‚(:‰ ßçŠÓݱҳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.132' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÑUÍD´"‚(:¹  ßçŠÓ¸öó³ disk.drbd2041.out‚0* proc.memory.percent‚11.863136291503906 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1550.out‚0 disk.sda.out‚ 367.78515625 disk.drbd1673.in‚0% disk./.usageGB‚1260.8545455932617 disk.rbd1.in‚0 disk.drbd1965.in‚0 network.recv‚932983 disk.drbd2111.in‚0 disk.drbd1289.out‚0 cpu‚25.32768455607782" disk.dm-0.out‚43131.4990234375! proc.memory.rssMB‚ 2915.484375 disk.rbd1.out‚ 74.3193359375 disk.drbd2118.out‚0% disk.nvme0n1.out‚43131.4990234375 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 network.sent‚ 1.3828013e+07) disk./.usagePercent‚70.51358670200042 disk.nvme0n1.in‚ 96.47265625 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 367.78515625 disk.drbd1965.out‚0 disk.drbd2078.out‚0$ memory_percent‚92.15121269226074) proc.memory.availableMB‚ 1928.91796875 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0ªO‹Ÿµ"‚(:” îçŠÓݚܳ gpu.0.gpu‚94.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.964' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.98799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0],6—=¶"‚(:²  îçŠÓä“ÿ³ disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1928.62109375 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.863136291503906 network.recv‚934050 disk.drbd1356.out‚0 cpu‚25.327637644118425 network.sent‚ 1.3831065e+07 disk.drbd2111.out‚0 disk.sda.out‚ 368.01953125 disk.dm-0.in‚ 96.47265625 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.dm-0.out‚ 43131.859375% disk./.usageGB‚1260.8544845581055 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0! proc.memory.rssMB‚ 2915.484375 disk.sda2.out‚ 368.01953125 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1938.out‚0$ memory_percent‚92.15242067972818) disk./.usagePercent‚70.51358328859496 disk.drbd1828.out‚0! disk.nvme0n1.out‚ 43131.859375 disk.drbd1510.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 74.3193359375 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1965.in‚0k|Üù·"‚(:’ ýçŠÓ­´½³ gpu.0.gpu‚69.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.83999999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;u‘€9¸"‚(:®  ýçŠÓŸ³á³ disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.dm-0.out‚ 43132.03125 disk.drbd1965.in‚0 disk.nvme0n1.out‚ 43132.03125 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1510.out‚0 network.recv‚935172 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd2041.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 74.5615234375 disk.sda.in‚0.03125 disk.sda2.out‚ 368.3046875 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1928.62109375! proc.memory.rssMB‚ 2915.484375 disk.drbd1285.in‚0 disk.sda2.in‚0.03125 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0$ memory_percent‚92.15242067972818% disk./.usageGB‚1260.8545036315918 disk.drbd1965.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 368.3046875* proc.memory.percent‚11.863136291503906 disk.drbd1931.out‚0 disk.drbd1938.in‚0 network.sent‚ 1.3834156e+07) disk./.usagePercent‚70.51358435528417 disk.dm-0.in‚ 96.47265625 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2118.out‚0 cpu‚25.327574557391955­?£Ÿ¹"‚(:” ŒèŠÓÉÌÙ³ gpu.0.gpu‚77.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚300.443' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.14766666666665 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÓbáHº"‚(:½  ŒèŠÓÖð÷³ disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 368.66796875 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 368.66796875 cpu‚25.32925665016319) disk./.usagePercent‚70.51358222190575 disk.nvme0n1.in‚ 96.47265625 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1924.29296875 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd1356.in‚0& disk.nvme0n1.out‚43132.46533203125 disk.drbd1550.out‚0 network.sent‚ 1.3834222e+07 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2919.58984375% disk./.usageGB‚1260.8544654846191 disk.drbd1938.out‚0 network.recv‚935238# disk.dm-0.out‚43132.46533203125 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0$ memory_percent‚92.17003186543783 disk.drbd2078.out‚0 disk.rbd1.out‚ 74.5615234375 disk.drbd1584.in‚0* proc.memory.percent‚11.879841486612957`ÝG4»"/ _wandbruntime‚14289  _runtime‚14289:¿>° ¼"’v >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7213400602340698 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1591169536113739 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3244840204715729 *jet/accuracy_1_of_1‚0.3606431186199188 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5436222553253174 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8311371803283691 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7321427464485168 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3557913899421692 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6456233859062195 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6566473245620728 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6656227111816406 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3480982184410095 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333730697632 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5731743574142456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6945368647575378 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3487072288990021 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 )jet/accuracy_2_of_2‚0.167352095246315 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9925467371940613 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6025001406669617 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490260243415833 'particle/f_score‚0.6542544960975647 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4476780593395233 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33635759353637695 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 /particle/accuracy_3_of_4‚0.9948979616165161 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6132605671882629 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6899813413619995 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27701979875564575 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6609710454940796 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6237789988517761 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26521286368370056 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8114048838615417 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08001627773046494 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9217467308044434 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7639421820640564 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 >qÃh.Purity/1FRt*SRqqt0FBt/FRt_purity‚0.28015825152397156 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1458333283662796 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2918907105922699 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8541358113288879 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8519379496574402 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20962412655353546 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3724950850009918 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8747974038124084 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3876001238822937 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 *jet/accuracy_2_of_3‚0.5963822603225708 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891200065612793 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6994951963424683 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014005602337419987 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6391811966896057 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014005602337419987 *jet/accuracy_1_of_4‚0.9698782563209534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2891879081726074 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911409616470337 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34954220056533813 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14355435967445374 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26348334550857544 particle/accuracy_3_of_3‚1 $ _timestamp‚1.784853525864338e+09 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27628377079963684 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 +jet/accuracy_4_of_4‚0.02295798622071743 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3356800079345703 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.994532585144043 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4916395843029022 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15590094029903412 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6439560651779175 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46252816915512085 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25013989210128784 _runtime‚14289.222452805 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07731765508651733 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 +particle/sensitivity‚0.6448173522949219 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9457439184188843 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962924122810364 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5929095149040222 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.796652615070343 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6265257000923157 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12570719420909882 +particle/specificity‚0.9108540415763855 epoch‚27 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6576312184333801 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.872651994228363 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6222836375236511 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 _step‚651 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7637160420417786 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7910160422325134 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.062476955354213715 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7105633020401001 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8757354021072388 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0415148101747036 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.715053141117096 +jet/accuracy_3_of_4‚0.11857572942972183 6validation_average_jet_accuracy‚0.3960117697715759 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7420009970664978 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8339517712593079 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33635759353637695 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6814791560173035 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05042111501097679 )loss/val_total_loss‚7.972764731640007 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 (particle/accuracy‚0.8537772297859192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6789833903312683 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9270833134651184 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7212515473365784 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 *jet/accuracy_1_of_2‚0.7814720273017883 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.758747935295105 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17240561544895172 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9315678477287292 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027118008583784103 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15967635810375214 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 /particle/accuracy_4_of_4‚0.9761282205581665 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974095225334167 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025524018332362175 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25501757860183716 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6259912848472595 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6357022523880005 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.621711790561676 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7207475900650024 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03405795618891716 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7985043525695801 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33886057138442993 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.747772753238678 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24459262192249298 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47727271914482117 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4819263815879822 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03624530881643295 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1458333283662796 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4529585838317871 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2674238979816437 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7988469004631042 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6977947950363159 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37469062209129333 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21294023096561432 trainer/global_step‚7839 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37184906005859375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32759299874305725 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6265257000923157 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8757354021072388 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7659448981285095 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077ëx•;½"—v :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32759299874305725 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6265257000923157 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8757354021072388 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7659448981285095 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7213400602340698 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1591169536113739 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3244840204715729 *jet/accuracy_1_of_1‚0.3606431186199188 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5436222553253174 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8311371803283691 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7321427464485168 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3557913899421692 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6456233859062195 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6566473245620728 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6656227111816406 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3480982184410095 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333730697632 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5731743574142456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6945368647575378 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3487072288990021 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 )jet/accuracy_2_of_2‚0.167352095246315 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9925467371940613 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6025001406669617 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490260243415833 'particle/f_score‚0.6542544960975647 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4476780593395233 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33635759353637695 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 /particle/accuracy_3_of_4‚0.9948979616165161 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6132605671882629 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6899813413619995 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27701979875564575 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6609710454940796 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6237789988517761 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26521286368370056 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8114048838615417 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08001627773046494 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9217467308044434 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7639421820640564 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.28015825152397156 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1458333283662796 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2918907105922699 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8541358113288879 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8519379496574402 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20962412655353546 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3724950850009918 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8747974038124084 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3876001238822937 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 *jet/accuracy_2_of_3‚0.5963822603225708 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891200065612793 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6994951963424683 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014005602337419987 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6391811966896057 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014005602337419987 *jet/accuracy_1_of_4‚0.9698782563209534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2891879081726074 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911409616470337 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34954220056533813 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14355435967445374 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26348334550857544 particle/accuracy_3_of_3‚1 $ _timestamp‚1.784853525864338e+09 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27628377079963684 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 +jet/accuracy_4_of_4‚0.02295798622071743 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3356800079345703 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.994532585144043 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4916395843029022 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15590094029903412 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6439560651779175 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46252816915512085 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25013989210128784 _runtime‚14289.222452805 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07731765508651733 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 +particle/sensitivity‚0.6448173522949219 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9457439184188843 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962924122810364 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5929095149040222 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.796652615070343 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6265257000923157 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12570719420909882 +particle/specificity‚0.9108540415763855 epoch‚27 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6576312184333801 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.872651994228363 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6222836375236511 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 _step‚651 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7637160420417786 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7910160422325134 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.062476955354213715 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7105633020401001 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8757354021072388 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0415148101747036 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.715053141117096 +jet/accuracy_3_of_4‚0.11857572942972183 6validation_average_jet_accuracy‚0.3960117697715759 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7420009970664978 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8339517712593079 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33635759353637695 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6814791560173035 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05042111501097679 )loss/val_total_loss‚7.972764731640007 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 (particle/accuracy‚0.8537772297859192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6789833903312683 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9270833134651184 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7212515473365784 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 *jet/accuracy_1_of_2‚0.7814720273017883 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.758747935295105 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17240561544895172 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9315678477287292 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027118008583784103 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15967635810375214 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 /particle/accuracy_4_of_4‚0.9761282205581665 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974095225334167 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025524018332362175 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25501757860183716 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6259912848472595 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6357022523880005 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.621711790561676 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7207475900650024 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03405795618891716 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7985043525695801 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33886057138442993 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.747772753238678 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24459262192249298 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47727271914482117 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4819263815879822 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03624530881643295 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1458333283662796 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4529585838317871 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2674238979816437 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7988469004631042 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6977947950363159 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37469062209129333 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21294023096561432 trainer/global_step‚7839 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37184906005859375‹%Oœ¾"‚(:‘ ›èŠÓþêѳ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚187.41' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚62.470000000000006 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ}±C¿"‚(:¸  ›èŠÓ¯Ëò³ disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.sda.out‚ 368.96875 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.901330947875977 cpu‚25.330901149237295 disk.drbd2111.out‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 network.recv‚935238% disk./.usageGB‚1260.8549537658691 disk.drbd1552.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd2118.in‚0$ memory_percent‚92.19174385070801 network.sent‚ 1.3834222e+07 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.rbd1.out‚ 74.7255859375 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51360952914939 disk.drbd1673.out‚0) proc.memory.availableMB‚ 1918.95703125# proc.memory.rssMB‚ 2924.87109375 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 96.47265625 disk.sda2.out‚ 368.96875# disk.dm-0.out‚43132.66845703125& disk.nvme0n1.out‚43132.66845703125 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0¥ä÷™4À"/ _wandbruntime‚14305  _runtime‚14305•»ÍÁÆÁ"À % lr-AdamW/pg1‚4.90984981772388e-05 % lr-AdamW/pg2‚4.90984981772388e-05 trainer/global_step‚7849 % _timestamp‚1.7848535417855036e+09 _runtime‚14305.142754528 _step‚652«ç{žËÂ"Å % _timestamp‚1.7848535417855036e+09 _runtime‚14305.142754528 _step‚652 % lr-AdamW/pg1‚4.90984981772388e-05 % lr-AdamW/pg2‚4.90984981772388e-05 trainer/global_step‚7849ŒQöÅ4Ã"/ _wandbruntime‚14305  _runtime‚14305€ d Ä"ƒP L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68189454 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8181145260988 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2269418457007104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68189460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72905803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚618466522307584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20921679 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8179800649702 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚14305.14794967 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141095257 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2261224936650752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848535417865467e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1¸E¥371609600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44852469 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2261224962824192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2269418457007104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72915867 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2269419828616704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44852732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36587008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68189454 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23930790 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2268513894785356 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2268512524746102 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _step‚653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2269419828616704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2261224936650752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚618466154216448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2261224962824192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10021111534080 trainer/global_step‚7849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404678144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚628487229163520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10021074947072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8193520356352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20921684 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141105327 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23931048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8194865792512 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _step‚653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2269419828616704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2261224936650752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚618466154216448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2261224962824192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10021111534080 trainer/global_step‚7849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404678144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚628487229163520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10021074947072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8193520356352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20921684 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141105327 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23931048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8194865792512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20921679 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8179800649702 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚14305.14794967 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141095257 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2261224936650752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848535417865467e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44852469 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2261224962824192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2269418457007104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72915867 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2269419828616704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44852732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36587008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68189454 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23930790 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2268513894785356 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2268512524746102 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10022426151424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 trainer/global_step‚7849 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72914885 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2269699760422400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2268792347321518 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2268793717360884 _runtime‚14306.61604155 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141122790 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚618545108673536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44858158 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23933839 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2269699760422400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2261503872114688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44858423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23934099 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141122790 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2261503872114688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8195888307712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404670976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8182166062740 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72924963 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2261503845941248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68197821 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141112706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628567130153984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10022389571584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8194542864384 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2269698388805632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚654 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2261503845941248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398NÊE~äÈ"‘N \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8195888307712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2261503845941248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68197821 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚618544740582400 $ _timestamp‚1.784853543258401e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72924963 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68197827 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10022426151424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 trainer/global_step‚7849 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72914885 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2269699760422400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2268792347321518 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2268793717360884 _runtime‚14306.61604155 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141122790 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚618545108673536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44858158 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23933839 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2269699760422400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2261503872114688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44858423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23934099 XIDeviceStatsMonitor.on_train_batchzúÄ3_end/reserved_bytes.small_pool.allocated‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141122790 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2261503872114688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8195888307712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404670976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8182166062740 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72924963 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2261503845941248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68197821 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141112706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628567130153984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10022389571584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8194542864384 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2269698388805632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚654 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚14392.212696771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68607810 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21053679 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8230904532510 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2283415111088128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141979629 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8246055348736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10084670844416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2274272812785200 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2282503717317710 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68607804 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8246055348736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2275170427348992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21053684 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632480143358464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2275170401175552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73371819 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2283415111088128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10084728402944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2275170401175552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68607804 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45132637 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24079306 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622395840605184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68607810 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8244709912576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚657 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2275170427348992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622395472514048 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2282505087356964 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181K#5ˆ Ý"‰P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73361755 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8232249143796 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73371819 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2283415111088128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10084728402944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2275170401175552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68607804 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45132637 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24079306 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622395840605184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68607810 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8244709912576 F@DeviceStatsMonitor.oã 8‡n_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _step‚657 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2275170427348992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622395472514048 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2282505087356964 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7848536288524659e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73361755 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2283416482697728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73371819 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2274272838213168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24078958 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚14392.212696771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68607810 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21053679 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8230904532510 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2283415111088128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141979629 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8246055348736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10084670844416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2274272812785200 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2282503717317710 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68607804 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8246055348736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2275170427348992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21053684 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632480143358464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2275170401175552 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2275449310466048 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8247077863936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21056319 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8231925334150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2275449336639488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2274551639986944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73370837 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24082234 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2282784909932492 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8233269945548 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8245732420608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10085972288512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚658 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2275449336639488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141997092 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45138558 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141997092 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚632559979908608 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 % _timestamp‚1.7848536303281765e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141987008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68616171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8245732420608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45138208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2283696414503424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68616177 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622474058880000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2282783539893126 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21056324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181s–f›'à"“N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2282783539893126 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21056324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2275449310466048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73370837 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2275449310466048 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8247077863936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21056319 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8231925334150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2275449336639488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2274551639986944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73370837 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24082234 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2282784909932492 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8233269945548 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8245732420608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10085972288512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚658 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2275449336639488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141997092 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45138558 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141997092 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚632559979908608 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 % _timestamp‚1.7848536303281765e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141987008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68616171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8245732420608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45138208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2283696414503424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68616177 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622474058880000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0’ºMÖ4á"/ _wandbruntime‚14393  _runtime‚14393܃ž`â" /loss/FRt2/assignment_loss‚0.640142255761091 3loss/SRqqt2/assignment_loss‚0.46666424163373943 1loss/SRqqt2/detection_loss‚0.6406050146376405 0loss/FBt1/detection_loss‚0.48198464196739776 $loss/total_loss‚7.62415745090564 epoch‚28 % _timestamp‚1.7848536303633525e+09 _step‚659 /loss/FRt1/assignment_loss‚1.609645568384913 3loss/SRqqt1/assignment_loss‚0.49378466144406846 .loss/FRt2/detection_loss‚1.497554078921947 trainer/global_step‚7899 /loss/FBt2/assignment_loss‚0.024641593106954 /loss/FBt2/detection_loss‚0.3296946095593739 _runtime‚14393.720559876 2loss/FBt1/assignment_loss‚4.66685262258631e-05 /loss/FRt1/detection_loss‚0.5273126552629099 1loss/SRqqt1/detection_loss‚0.9120814616993799‚Q=ã ã"† /loss/FRt1/detection_loss‚0.5273126552629099 1loss/SRqqt1/detection_loss‚0.9120814616993799 /loss/FRt2/assignment_loss‚0.640142255761091 3loss/SRqqt2/assignment_loss‚0.46666424163373943 1loss/SRqqt2/detection_loss‚0.6406050146376405 0loss/FBt1/detection_loss‚0.48198464196739776 $loss/total_loss‚7.62415745090564 epoch‚28 % _timestamp‚1.7848536303633525e+09 _step‚659 /loss/FRt1/assignment_loss‚1.609645568384913 3loss/SRqqt1/assignment_loss‚0.49378466144406846 .loss/FRt2/detection_loss‚1.497554078921947 trainer/global_step‚7899 /loss/FBt2/assignment_loss‚0.024641593106954 /loss/FBt2/detection_loss‚0.3296946095593739 _runtime‚14393.720559876 2loss/FBt1/assignment_loss‚4.66685262258631e-05“í²"b“ä"‚(:ˆ „éŠÓ³ª³³ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚262.482' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚87.494 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª[ÆËFå"‚(:»  „éŠÓ՗ѳ disk.rbd1.out‚ 75.2802734375& disk.nvme0n1.out‚43136.90771484375 disk.drbd1817.out‚0 disk.drbd1510.out‚0 network.sent‚ 1.3923565e+07 disk.sda2.out‚ 371.76953125 disk.drbd2118.in‚0$ memory_percent‚92.19226837158203 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1289.out‚0) disk./.usagePercent‚70.51361827600087 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.dm-0.in‚ 96.47265625 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.901330947875977 disk.sda.in‚0.03125 disk.drbd1673.in‚0 cpu‚25.330538753482077 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0' proc.memory.availableMB‚ 1918.828125$ disk./.usageGB‚1260.855110168457 disk.nvme0n1.in‚ 96.47265625 disk.drbd1550.out‚0 network.recv‚941279 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.sda.out‚ 371.76953125 disk.drbd2111.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚43136.90771484375 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0@gÙœæ"‚(:‘ “éŠÓ×ÅÙ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.16' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.053333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ââÓdEç"‚(:º  “éŠÓŒ£ù³ disk.drbd2111.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.sda2.out‚ 372.23046875 cpu‚25.33048858935316) disk./.usagePercent‚70.51367289048812 disk.drbd2041.in‚0 disk.drbd1552.in‚0# disk.dm-0.out‚43137.39697265625 disk.drbd1550.out‚0* proc.memory.percent‚11.901330947875977 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 372.23046875 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 75.4443359375 disk.drbd1817.out‚0 disk.drbd1226.in‚0' proc.memory.availableMB‚ 1919.078125 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 96.47265625& disk.nvme0n1.out‚43137.39697265625 network.sent‚ 1.3923673e+07 network.recv‚941387 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0$ disk./.usageGB‚1260.856086730957 disk.sda1.in‚0$ memory_percent‚92.19125111897787# proc.memory.rssMB‚ 2924.87109375 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0ûß@ÚÃè"‚(:’ ¢éŠÓû‡¸³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatGÃèHÚts‚296.89' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.96333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­GµGé"‚(:¼  ¢éŠÓœ‡×³ disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda.out‚ 372.50390625 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 75.4443359375 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 96.47265625# disk.dm-0.out‚43137.55322265625 disk.drbd2078.out‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚43137.55322265625 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 372.50390625 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 cpu‚25.33038881404855 network.recv‚942699 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.901330947875977 disk.drbd1817.out‚0 network.sent‚ 1.3931911e+07) disk./.usagePercent‚70.51367395717733 disk.nvme0n1.in‚ 96.47265625 disk.drbd1510.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.19325383504233( proc.memory.availableMB‚ 1918.5859375 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0% disk./.usageGB‚1260.8561058044434 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd2041.in‚0݉³žê"‚(:“ ±éŠÓî…´³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚226.219' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.40633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05QŸë<ë"‚(:±  ±éŠÓù™Ò³ disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.sda2.out‚ 372.84375 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 372.84375 disk.drbd1673.in‚0$ memory_percent‚92.19239552815756) disk./.usagePercent‚70.51367395717733 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 75.4765625 disk.drbd1285.out‚0 disk.drbd1510.in‚0" disk.dm-0.out‚43137.8681640625 disk.drbd1356.in‚0 network.recv‚942765% disk./.usageGB‚1260.8561058044434 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1931.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 network.sent‚ 1.3931977e+07* proc.memory.percent‚11.901330947875977% disk.nvme0n1.out‚43137.8681640625 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0' proc.memory.availableMB‚ 1918.796875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd2041.in‚0 cpu‚25.330340199581034Öîb‘ì"‚(:† ÀéŠÓ¤»¹³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.675' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zè@¾Aí"‚(:¶  ÀéŠÓù¬Û³ disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.sda2.out‚ 373.0390625 cpu‚25.330274051971017 disk.dm-0.in‚ 96.47265625 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2924.87109375 disk.rbd1.out‚ 75.4765625& disk.nvme0n1.out‚43138.07177734375 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1550.in‚0 network.sent‚ 1.4002988e+07 disk.drbd1550.out‚0 disk.sda1.out‚0$ memory_percent‚92.19239552815756 disk.drbd1285.in‚0 disk.drbd1931.out‚0$ disk./.usageGB‚1260.856113433838 disk.nvme0n1.in‚ 96.47265625 disk.drbd1226.out‚0 disk.drbd1673.out‚0# disk.dm-0.out‚43138.07177734375 network.recv‚946836) disk./.usagePercent‚70.51367438385302 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.901330947875977 disk.drbd1552.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 373.0390625 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0' proc.memory.availableMB‚ 1918.796875 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0u6”î"‚(:‰ ÏéŠÓˆþá³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.766' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U“žCï"‚(:¸  ÏéŠÓ¾à€´ disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 96.47265625 disk.drbd1226.out‚0 cpu‚25.330207226266136 network.sent‚ 1.4003788e+07( disk./.usagePercent‚70.5136733171638 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 75.7265625 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0% disk./.usageGB‚1260.8560943603516 disk.drbd1931.in‚0 disk.dm-0.in‚ 96.47265625 disk.drbd1965.in‚0$ memory_percent‚92.19037691752115 disk.drbd1510.in‚0 disk.sda.out‚ 373.47265625 disk.sda2.out‚ 373.47265625 disk.drbd1289.in‚0" disk.dm-0.out‚43138.6962890625 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.901330947875977 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1285.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1919.29296875 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2924.87109375 proc.cpu.threads‚17 network.recv‚946968 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚43138.6962890625â8À4ð"/ _wandbruntime‚14479  _runtime‚14479õ¿âÇñ"Á & lr-AdamW/pg2‚4.777772379548286e-05 trainer/global_step‚7949 % _timestamp‚1.7848537159223988e+09 _runtime‚14479.27983435 _step‚660 & lr-AdamW/pg1‚4.777772379548286e-05|H»±Ìò"Æ & lr-AdamW/pg1‚4.777772379548286e-05 & lr-AdamW/pg2‚4.777772379548286e-05 trainer/global_step‚7949 % _timestamp‚1.7848537159223988e+09 _runtime‚14479.27983435 _step‚660”“ E4ó"/ _wandbruntime‚14479  _runtime‚14479çd¥| (ô"„P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142843861 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24226387 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2289115891873792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2289115891873792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8283353026604 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚14479.280705338 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8295899468800 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2288212926901968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 % _timestamp‚1.7848537159232028e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2297411765169152 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2296494909889318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21185684 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73817707 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45412417 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69026160 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2297413136778752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45412066 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297244904960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8282008415318 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21185679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69026160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2288212901474000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2297411765169152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142853931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2289115865700352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7949 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73827771 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2296496279928572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8295899468800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10147297169920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8297244904960 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10147248000000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24226733 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248郩(õ"‰P [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8297244904960 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10147248000000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24226733 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142843861 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24226387 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2289115891873792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2289115891873792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8283353026604 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚14479.280705338 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8295899468800 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2288212926901968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 % _timestamp‚1.7848537159232028e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2297411765169152 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2296494909889318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21185684 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73817707 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45412417 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69026160 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2297413136778752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45412066 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297244904960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8282008415318 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21185679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69026160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2288212901474000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2297411765169152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142853931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2289115865700352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7949 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73827771 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2296496279928572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8295899468800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10147297169920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755•ˆ”Û‰4ö"/ _wandbruntime‚14480  _runtime‚14480OÅݺ¤÷"ŽN REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73836867 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2289394801164288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24229688 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2296774732464734 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69034527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24229339 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2289394774990848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚636551870414848 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _step‚662 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142871394 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚14480.757838513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21188319 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2289394774990848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21188324 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2297693068584448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8298267420160 cODeviceStatsMonitor.on_train_batch_ÐUGÃpend/inactive_split_bytes.large_pool.allocated‚626403745268736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10148548691456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8298267420160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8284373828356 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8283029216958 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73836867 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2289394801164288 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848537174002042e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69034527 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142861310 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2288491703247776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69034521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45418012 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2288491728675744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142871394 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8296921976832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45417658 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73836867 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2289394801164288 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848537174002042e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69034527 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142861310 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2288491703247776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69034521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45418012 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2288491728675744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142871394 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8296921976832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45417658 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24229688 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2296774732464734 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69034527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24229339 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2289394774990848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚636551870414848 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _step‚662 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142871394 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚14480.757838513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21188319 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2289394774990848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21188324 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2297693068584448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8298267420160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626403745268736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10148548691456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8298267420160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8284373828356 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8283029216958 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EAÿ"‚(:¶  íéŠÓסֳ disk.dm-0.out‚43139.35546875 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1931.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 75.7587890625 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125# disk.nvme0n1.out‚43139.35546875 disk.drbd1817.in‚0 disk.drbd1552.out‚0 network.sent‚ 1.407318e+07 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0# memory_percent‚92.1896775563558) proc.memory.availableMB‚ 1919.46484375 network.recv‚959093 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.sda2.out‚ 374.29296875 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.901330947875977) disk./.usagePercent‚70.51366905040699 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1931.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 98.87109375% disk./.usageGB‚1260.8560180664062 disk.drbd1289.in‚0 disk.sda.out‚ 374.29296875 disk.drbd1584.out‚0 disk.drbd1673.out‚0 cpu‚25.330077311619082 proc.cpu.threads‚17AB­Ð=€#‚(:²  üéŠÓ¤µÓ³ disk.drbd1550.in‚0 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd2078.out‚0 cpu‚25.330012042689916) disk./.usagePercent‚70.51367118378539 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0$ disk./.usageGB‚1260.856056213379 disk.drbd2111.in‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.901330947875977 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 network.sent‚ 1.40742e+07 disk.drbd1584.out‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 98.87109375 disk.drbd1226.out‚0 disk.rbd1.out‚ 75.7587890625" disk.nvme0n1.out‚ 43139.4921875 disk.sda.out‚ 374.59765625) proc.memory.availableMB‚ 1919.30859375 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.sda2.out‚ 374.59765625 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.dm-0.out‚ 43139.4921875 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0# memory_percent‚92.1903133392334 network.recv‚960170 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0gÀRž#‚(:“ üéŠÓ‚©²³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.442' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.14733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?pŸ‚#‚(:” ‹êŠÓÓþì³ gpu.0.gpu‚72.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.107' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.36900000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–µÆã:ƒ#‚(:¯  ‹êŠÓÒ‰´ disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0$ memory_percent‚92.19131469726562 disk.sda.out‚ 375.33984375 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1356.out‚0 disk.rbd1.out‚ 75.9228515625 disk.drbd1510.in‚0 disk.drbd1552.out‚0% proc.memory.availableMB‚ 1919.0625 network.recv‚961067$ disk./.usageGB‚1260.856113433838 disk.drbd2118.in‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51367438385302 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1510.out‚0 disk.sda.in‚0.03125 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 375.33984375 disk.drbd1226.out‚0 network.sent‚ 1.4076648e+07 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2041.out‚0! disk.nvme0n1.out‚ 43139.921875 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.drbd1828.out‚0 cpu‚25.329979597807117 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.901330947875977 disk.drbd1938.out‚0 disk.dm-0.out‚ 43139.921875ÐzvZž„#‚(:“ šêŠÓÄÌÒ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚280.625' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.54166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝþÌ7…#‚(:¬  šêŠÓåÉõ³ disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.rbd1.out‚ 75.9228515625 disk.nvme0n1.in‚ 98.87109375 disk.nvme0n1.out‚ 43140.09375) disk./.usagePercent‚70.51357752847326% disk./.usageGB‚1260.8543815612793 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 98.87109375* proc.memory.percent‚11.901330947875977 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.dm-0.out‚ 43140.09375 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 375.8671875 disk.drbd1510.in‚0 disk.drbd1356.in‚0 disk.sda2.in‚0.03125$ memory_percent‚92.19131469726562 network.recv‚962075 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 375.8671875 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0% proc.memory.availableMB‚ 1919.0625 disk.drbd1584.out‚0 network.sent‚ 1.4079635e+07 disk.drbd2111.out‚0 disk.drbd1584.in‚0 cpu‚25.329932181667157# proc.memory.rssMB‚ 2924.87109375 disk.drbd1550.in‚0uàW—ž†#‚(:“ ©êŠÓ©Óë³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚265.432' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.47733333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¬4~C‡#‚(:¸  ©êŠÓ·Ð‹´ disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.dm-0.in‚ 98.87109375% proc.memory.availableMB‚ 1919.0625 disk.drbd1356.in‚0* proc.memory.percent‚11.901330947875977 disk.drbd1931.out‚0% disk./.usageGB‚1260.8546333312988 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.sda.out‚ 376.29296875 disk.drbd2078.out‚0 cpu‚25.329883686409204 network.recv‚963004 disk.drbd1828.in‚0& disk.nvme0n1.out‚43140.35107421875 proc.cpu.threads‚17) disk./.usagePercent‚70.51359160877075 disk.sda1.out‚0$ memory_percent‚92.19131469726562 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 75.955078125 network.sent‚ 1.408262e+07 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚43140.35107421875 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 376.29296875 disk.drbd1931.in‚0 disk.drbd1289.out‚0«Š¸`4ˆ#/ _wandbruntime‚14565  _runtime‚14565¯È‰# _runtime‚14565.97883318 _step‚664 ' lr-AdamW/pg1‚4.7119414218388045e-05 ' lr-AdamW/pg2‚4.7119414218388045e-05 trainer/global_step‚7999 $ _timestamp‚1.784853802621581e+09ãüßÿÍŠ#Ç ' lr-AdamW/pg2‚4.7119414218388045e-05 trainer/global_step‚7999 $ _timestamp‚1.784853802621581e+09 _runtime‚14565.97883318 _step‚664 ' lr-AdamW/pg1‚4.7119414218388045e-05˜_˜Šå4‹#/ _wandbruntime‚14565  _runtime‚14565'×JiŒ#„P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2311409790859776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2311409790859776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74273659 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45691412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74283723 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2303061330225152 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2311408419250176 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2303061330225152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74273659 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24374097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2302153015590768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8334456909412 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2311408419250176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10209821748224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143718163 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2310486102460926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248‡|– (#‰P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2303061330225152 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2311408419250176 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2303061330225152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74273659 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24374097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2302153015590768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8334456909412 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2311408419250176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10209821748224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143718163 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2310486102460926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2311409790859776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2311409790859776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74273659 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45691412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74283723 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005™SÉg»4Ž#/ _wandbruntime‚14567  _runtime‚14567*^Ò'#ŽN F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚14567.456408061 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640544137147392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143735612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45697323 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143745696 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45696956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21320324 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69452877 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8334133099766 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74292819 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2303340265689088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69452877 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8335477711164 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24376999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640543713602048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2311688351048704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8348111533056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21320319 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 trainer/global_step‚7999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚666 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143745696 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2311689722665472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69452871 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 % _timestamp‚1.7848538040987668e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚630333063566336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2311689722665472 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8348111533056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69452871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2302431791936576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚20050"–å#“N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2303340265689088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69452877 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8335477711164 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24376999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640543713602048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2311688351048704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8348111533056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21320319 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 trainer/global_step‚7999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚666 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143745696 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2311689722665472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69452871 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 % _timestamp‚1.7848538040987668e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚630333063566336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2311689722665472 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8348111533056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69452871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2302431791936576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚14567.456408061 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640544137147392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143735612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45697323 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143745696 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45696956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21320324 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69452877 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8334133099766 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74292819 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248šO›úR4‘#/ _wandbruntime‚14567  _runtime‚14567GÖR( ’#‡ epoch‚28 0loss/FBt2/detection_loss‚0.49712365507426215 %loss/total_loss‚8.001927086957254 2loss/SRqqt1/assignment_loss‚0.4330374030266809 2loss/SRqqt2/assignment_loss‚0.4847429876724818 2loss/FBt2/assignment_loss‚0.016254674693229967 /loss/FRt1/detection_loss‚0.5379453375481807 1loss/SRqqt1/detection_loss‚0.9007982670173761 /loss/FBt1/detection_loss‚0.6800218150019646 0loss/FRt1/assignment_loss‚1.6538093278585082 3loss/FBt1/assignment_loss‚0.0006775812654211677 /loss/FRt2/detection_loss‚1.4113661640860289 trainer/global_step‚7999 % _timestamp‚1.7848538041350834e+09 _runtime‚14567.492284708 _step‚667 0loss/FRt2/assignment_loss‚0.7859664109368988 1loss/SRqqt2/detection_loss‚0.6001834627762207X™¥Ê“#Œ _step‚667 0loss/FRt2/assignment_loss‚0.7859664109368988 1loss/SRqqt2/detection_loss‚0.6001834627762207 epoch‚28 0loss/FBt2/detection_loss‚0.49712365507426215 %loss/total_loss‚8.001927086957254 2loss/SRqqt1/assignment_loss‚0.4330374030266809 2loss/SRqqt2/assignment_loss‚0.4847429876724818 2loss/FBt2/assignment_loss‚0.016254674693229967 /loss/FRt1/detection_loss‚0.5379453375481807 1loss/SRqqt1/detection_loss‚0.9007982670173761 /loss/FBt1/detection_loss‚0.6800218150019646 0loss/FRt1/assignment_loss‚1.6538093278585082 3loss/FBt1/assignment_loss‚0.0006775812654211677 /loss/FRt2/detection_loss‚1.4113661640860289 trainer/global_step‚7999 % _timestamp‚1.7848538041350834e+09 _runtime‚14567.492284708›+=U€Ÿ”#‚(:” ¸êŠÓ¥Àdz gpu.0.gpu‚37.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚336.524' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚112.17466666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íëvCH•#‚(:½  ¸êŠÓ°ãè³ disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.901330947875977 network.recv‚964042 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1918.66015625 network.sent‚ 1.4085641e+07 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125# disk.dm-0.out‚43141.11279296875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2924.87109375 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0% disk./.usageGB‚1260.8555068969727$ memory_percent‚92.19295183817545 disk.sda.out‚ 376.63671875 disk.sda2.out‚ 376.63671875 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1550.in‚0 cpu‚25.329853999319784 disk.sda1.in‚0& disk.nvme0n1.out‚43141.11279296875) disk./.usagePercent‚70.51364046313631 disk.drbd1965.in‚0 disk.rbd1.out‚ 76.134765625 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1510.in‚0ü¡‰ž–#‚(:“ ÇêŠÓ’ôä³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚270.301' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.10033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05‰AS1—#‚(:¦  ÇêŠÓÌÛ…´ disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.901330947875977 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 98.87109375 network.recv‚966682 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 76.298828125 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1965.in‚0( disk./.usagePercent‚70.5136722504746 disk.rbd1.in‚0 disk.sda.in‚0.03125 disk.sda.out‚377 disk.sda2.in‚0.03125 disk.drbd1550.in‚0% proc.memory.availableMB‚ 1918.5625 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd2111.out‚0$ memory_percent‚92.19334920247397% disk./.usageGB‚1260.8560752868652 disk.drbd1285.in‚0& disk.nvme0n1.out‚43141.61767578125 disk.drbd1938.in‚0# disk.dm-0.out‚43141.61767578125 network.sent‚ 1.4145728e+07 disk.drbd1356.out‚0 disk.drbd2111.in‚0 cpu‚25.329754212781403 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚377 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0ØÈqûž˜#‚(:“ ÖêŠÓԽγ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.204' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.40133333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ƒB™#‚(:·  ÖêŠÓäŒé³ network.recv‚967608 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.901330947875977 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.sda.out‚ 377.265625 disk.drbd2111.in‚0 disk.drbd2111.out‚0# disk.dm-0.out‚43141.89501953125 disk.drbd1226.in‚0) proc.memory.availableMB‚ 1918.55859375 cpu‚25.32970746397836 disk.sda2.out‚ 377.265625 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 98.87109375) disk./.usagePercent‚70.51365177004188 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd2118.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0# memory_percent‚92.1933650970459 disk.drbd1817.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0% disk./.usageGB‚1260.8557090759277 disk.drbd1289.in‚0 disk.rbd1.out‚ 76.298828125 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 98.87109375& disk.nvme0n1.out‚43141.89501953125 network.sent‚ 1.4148726e+07 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0Ççõ“š#‚(:ˆ åêŠÓ⑶³ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.008' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"mÛ±B›#‚(:·  åêŠÓʧԳ) disk./.usagePercent‚70.51365177004188 disk.drbd1226.out‚0 disk.sda.out‚ 377.70703125 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1510.out‚0 disk.drbd1552.in‚0' proc.memory.availableMB‚ 1918.546875% disk./.usageGB‚1260.8557090759277 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0$ disk.nvme0n1.out‚43142.236328125 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.901330947875977 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1550.in‚0 cpu‚25.32962648688876 disk.sda2.out‚ 377.70703125 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.19341278076172 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 network.recv‚968677 disk.sda.in‚0.03125 disk.dm-0.in‚ 98.87109375 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1673.in‚0 disk.drbd2041.out‚0 network.sent‚ 1.4151698e+07! disk.dm-0.out‚43142.236328125 disk.rbd1.out‚ 76.3310546875 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0ȃ²á“œ#‚(:ˆ ôêŠÓ‡ŸÐ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚291.759' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.253 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xª’C#‚(:¸  ôêŠÓ¯Íò³ disk.drbd1828.in‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2924.87109375 network.recv‚969774 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 377.9453125 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0$ memory_percent‚92.19241142272949 disk.sda.out‚ 377.9453125 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1918.79296875) disk./.usagePercent‚70.51364750328506 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.901330947875977 cpu‚25.329578302966652 network.sent‚ 1.4155262e+07 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 76.3310546875 disk.nvme0n1.in‚ 98.87109375 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.8556327819824 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.sda.in‚0.03125! disk.dm-0.out‚43142.509765625 disk.drbd2078.out‚0$ disk.nvme0n1.out‚43142.509765625 disk.drbd1938.out‚0 disk.drbd1356.in‚0åÂ,54ž#/ _wandbruntime‚14652  _runtime‚14652›ÕOàÈŸ# & lr-AdamW/pg1‚4.646260483132922e-05 & lr-AdamW/pg2‚4.646260483132922e-05 trainer/global_step‚8049 % _timestamp‚1.7848538895662794e+09 _runtime‚14652.923517473 _step‚668 einÍ #Ç & lr-AdamW/pg1‚4.646260483132922e-05 & lr-AdamW/pg2‚4.646260483132922e-05 trainer/global_step‚8049 % _timestamp‚1.7848538895662794e+09 _runtime‚14652.923517473 _step‚668œ“Y÷4¡#/ _wandbruntime‚14652  _runtime‚14652Ë!> (¢#ƒP ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2325406444940800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69862854 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21449679 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74739675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144602535 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45971542 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74729611 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45971887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚634183795497984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8398278581248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2316093078851600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144592465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚634183427406848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10272438138880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2316093104279568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2325405073331200 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2325405073331200 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2317006820923392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8384216180934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Õn¦£#ˆP TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74739675 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2324478665071788 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2317006794749952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8399624017408 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848538895673277e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69862860 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10272393163264 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644456233636864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69862860 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚340 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644455820570112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8399624017408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8398278581248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24522203 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74729611 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚14652.92795946 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21449684 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2324477295032534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8049 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10272438138880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requesteשãhd_bytes.large_pool.allocated‚2316093104279568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2325405073331200 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2325405073331200 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2317006820923392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8384216180934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2325406444940800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69862854 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21449679 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74739675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144602535 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45971542 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74729611 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45971887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚634183795497984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8398278581248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2316093078851600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144592465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚634183427406848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45977104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24525125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21452324 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2316371880625376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69871227 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8399301089280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚340 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2325686376746496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144619998 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74738693 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2317285730213888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2324758487647316 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2325686376746496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45977449 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2317285730213888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21452319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚8049 % _timestamp‚1.7848538910414748e+09 _step‚670 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10273697451008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10273652482560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2316371906053344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2317285704040448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2324757117607950 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112›c›—'¦#“N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10273652482560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2316371906053344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2317285704040448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2324757117607950 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8400646532608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144619998 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69871221 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24524785 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45977104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24525125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21452324 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2316371880625376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69871227 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8399301089280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚340 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2325686376746496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144619998 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74738693 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2317285730213888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2324758487647316 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2325686376746496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45977449 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2317285730213888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21452319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚8049 % _timestamp‚1.7848538910414748e+09 _step‚670 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10273697451008žU94§#/ _wandbruntime‚14654  _runtime‚14654yµžu ¨#„ 1loss/SRqqt1/detection_loss‚0.8547110445948426 _step‚671 0loss/FRt1/assignment_loss‚1.6290542458226935 /loss/FRt2/detection_loss‚1.3455216928140883 0loss/SRqqt2/detection_loss‚0.627901193344383 % _timestamp‚1.7848538910790594e+09 /loss/FBt2/detection_loss‚0.3953939786186676 %loss/total_loss‚7.808390015776647 epoch‚28 trainer/global_step‚8049 _runtime‚14654.436270044 3loss/SRqqt2/assignment_loss‚0.45759316390579435 3loss/FBt1/assignment_loss‚0.0008632141956074652 1loss/FBt2/assignment_loss‚0.07578914211927958 /loss/FRt1/detection_loss‚0.5345038827627857 /loss/FBt1/detection_loss‚0.6335307029579338 0loss/FRt2/assignment_loss‚0.7103332651302954 1loss/SRqqt1/assignment_loss‚0.543194489510275ÌøY©#‰ 0loss/FRt1/assignment_loss‚1.6290542458226935 /loss/FRt2/detection_loss‚1.3455216928140883 0loss/SRqqt2/detection_loss‚0.627901193344383 % _timestamp‚1.7848538910790594e+09 /loss/FBt2/detection_loss‚0.3953939786186676 %loss/total_loss‚7.808390015776647 epoch‚28 trainer/global_step‚8049 _runtime‚14654.436270044 3loss/SRqqt2/assignment_loss‚0.45759316390579435 3loss/FBt1/assignment_loss‚0.0008632141956074652 1loss/FBt2/assignment_loss‚0.07578914211927958 /loss/FRt1/detection_loss‚0.5345038827627857 /loss/FBt1/detection_loss‚0.6335307029579338 0loss/FRt2/assignment_loss‚0.7103332651302954 1loss/SRqqt1/assignment_loss‚0.543194489510275 1loss/SRqqt1/detection_loss‚0.8547110445948426 _step‚671ŸqiŠ‚Ÿª#‚(:” ƒëŠÓö¿©³ gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.855' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.61833333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–JôF«#‚(:»  ƒëŠÓÞÜÒ³ disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 76.4951171875 disk.drbd1226.out‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51370446448857 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8566513061523 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 98.87109375% disk.nvme0n1.out‚43143.3876953125 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1817.out‚0 cpu‚25.32953198571659 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0$ memory_percent‚92.19504992167154) proc.memory.availableMB‚ 1918.14453125 network.sent‚ 1.4157812e+07 disk.drbd1938.in‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 378.28515625 disk.drbd1931.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.901330947875977 disk.drbd2118.out‚0 disk.sda.out‚ 378.28515625# proc.memory.rssMB‚ 2924.87109375" disk.dm-0.out‚43143.3876953125 network.recv‚970756}d–¡ž¬#‚(:“ ’ëŠÓ´ÕÔ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚278.762' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.92066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bäVÄF­#‚(:»  ’ëŠÓÔô³ disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚43144.03662109375 disk.sda.out‚ 378.7109375 proc.cpu.threads‚17 disk.dm-0.in‚ 98.87109375 disk.drbd1356.out‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.8566284179688 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 76.4951171875 disk.sda1.out‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51370318446152 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1673.out‚0$ memory_percent‚92.19427108764648 cpu‚25.329449882416082# disk.dm-0.out‚43144.03662109375( proc.memory.availableMB‚ 1918.3359375 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 network.sent‚ 1.4218421e+07 network.recv‚973518 disk.sda2.out‚ 378.7109375 disk.nvme0n1.in‚ 98.87109375 disk.drbd1550.in‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.901330947875977 disk.drbd1226.in‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0»”- ž®#‚(:“ ¡ëŠÓ‘ìij gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚275.948' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.98266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô¡õã(¯#‚(:¸  ¡ëŠÓ’ç±´" disk.dm-0.out‚43144.9404296875 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1918.32421875# proc.memory.rssMB‚ 2924.87109375 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 379.3359375 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 network.sent‚ 1.4220916e+07) disk./.usagePercent‚70.51370318ë2—d446152 disk.drbd1510.out‚0 disk.drbd1552.in‚0 cpu‚25.32941816353919 disk.rbd1.in‚0 disk.drbd2118.in‚0# memory_percent‚92.1943187713623% disk.nvme0n1.out‚43144.9404296875 network.recv‚974580% disk./.usageGB‚1260.8566284179688 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.901330947875977 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 76.6435546875 disk.nvme0n1.in‚ 98.87109375 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.sda.out‚ 379.3359375¨Ì D°#‚(:¹  °ëŠÓåÈγ( proc.memory.availableMB‚ 1918.4921875 disk.drbd1828.out‚0$ memory_percent‚92.19363530476889 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0) disk./.usagePercent‚70.51370211777231 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.901330947875977 disk.drbd1510.in‚0 disk.drbd2118.in‚0 network.sent‚ 1.4224452e+07 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8566093444824 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 379.6484375 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 379.6484375 disk.drbd1931.in‚0" disk.dm-0.out‚43145.1279296875 disk.rbd1.out‚ 76.6435546875 disk.drbd1552.out‚0 network.recv‚975554 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda.in‚0.03125 cpu‚25.329357066379686 disk.nvme0n1.in‚ 98.87109375 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚43145.1279296875 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2924.87109375 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 proc.cpu.threads‚17oKwž±#‚(:“ °ëŠÓ ø³³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.26' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.08666666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^"v¨²#‚(:’ ¿ëŠÓüåÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.96' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.98666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»Íð?³#‚(:´  ¿ëŠÓÈ÷÷³ disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd2078.in‚0$ memory_percent‚92.19465255737305# proc.memory.rssMB‚ 2924.87109375 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0( proc.memory.availableMB‚ 1918.2421875 disk.rbd1.out‚ 76.9873046875 disk.drbd1965.in‚0 network.sent‚ 1.4226903e+07 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 cpu‚25.32927518556593) disk./.usagePercent‚70.51369571763709 disk.dm-0.in‚ 98.87109375" disk.dm-0.out‚43145.4443359375 disk.drbd1550.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.901330947875977% disk.nvme0n1.out‚43145.4443359375 disk.sda.in‚0.03125 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.sda2.out‚ 379.96875 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2118.out‚0 network.recv‚976496% disk./.usageGB‚1260.8564949035645 disk.drbd1226.out‚0 disk.sda.out‚ 379.96875 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0‰Ó+Èž´#‚(:“ ÎëŠÓþìä³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚271.842' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.61399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f`RFµ#‚(:»  ÎëŠÓÔËþ³ network.recv‚977463 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8566093444824 disk.rbd1.out‚ 76.9873046875 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 98.87109375 disk.drbd1965.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51370211777231 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 98.87109375( proc.memory.availableMB‚ 1918.2421875# proc.memory.rssMB‚ 2924.87109375 disk.drbd2078.out‚0% disk.nvme0n1.out‚43145.6513671875 disk.drbd1584.out‚0$ memory_percent‚92.19465255737305 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.sda1.out‚0 cpu‚25.329262197013566 disk.sda.out‚ 380.41796875 disk.sda2.out‚ 380.41796875 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0" disk.dm-0.out‚43145.6513671875 disk.drbd2078.in‚0 network.sent‚ 1.4229937e+07 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.901330947875977 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1828.in‚06m¢Ž4¶#/ _wandbruntime‚14739  _runtime‚14739"È·# & lr-AdamW/pg1‚4.580738172696882e-05 & lr-AdamW/pg2‚4.580738172696882e-05 trainer/global_step‚8099 % _timestamp‚1.7848539760996344e+09 _runtime‚14739.457882283 _step‚672È/ŽW͸#Ç _runtime‚14739.457882283 _step‚672 & lr-AdamW/pg1‚4.580738172696882e-05 & lr-AdamW/pg2‚4.580738172696882e-05 trainer/global_step‚8099 % _timestamp‚1.7848539760996344e+09 Ñ>Ër4¹#/ _wandbruntime‚14739  _runtime‚14739ryŽ (º#„P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46250340 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8435320063742 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848539761004455e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2338469857643396 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8436664675028 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10335094994432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2339403099021824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚673 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2339401727412224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2330033167540400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75185563 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145466767 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70281204 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75195627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648448208790016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8450813573632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2330952259274752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70281204 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2339403099021824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2330952285448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638113113795584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚341 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70281210 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10335043727360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2330952285448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75185563 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚14739.457987657 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145476837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24668320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70281210 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8450813573632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75195627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648447789431808 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2338468487604142 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2330952259274752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21581684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638112745704448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8449468137472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46249999]{dâ(»#‰P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2330952285448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75185563 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚14739.457987657 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145476837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24668320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70281210 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8450813573632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75195627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648447789431808 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2338468487604142 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2330952259274752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21581684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638112745704448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8449468137472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46249999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚336 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46250340 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8435320063742 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848539761004455e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2338469857643396 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8436664675028 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10335094994432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2339403099021824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚673 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2339401727412224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2330033167540400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75185563 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145466767 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70281204 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75195627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648448208790016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8450813573632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2330952259274752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70281204 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2339403099021824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2330952285448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638113113795584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚341 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70281210 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10335043727360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181¡G•€Ï4¼#/ _wandbruntime‚14740  _runtime‚14740¾–¥½#N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21584324 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2339683030827520 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2331231168565248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10336314427392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46255939 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75194645 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2331231194738688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8451836088832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2330311969314176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70289571 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75204723 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2331231194738688 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8437685476780 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2338749680218924 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2330311994742144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8451836088832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2338748310179558 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848539775771573e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8436340865382 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145494300 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚648528059557376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2331231168565248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638191700161536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46255601 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145494300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24671615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24671282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚14740.93479119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70289577 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145484216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75194645 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75204723 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚674 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10336359395840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚338 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚648527646497792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8450490645504 üX2'¾#’N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75204723 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2331231194738688 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8437685476780 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2338749680218924 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2330311994742144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8451836088832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2338748310179558 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848539775771573e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8436340865382 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145494300 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚648528059557376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2331231168565248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638191700161536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46255601 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145494300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24671615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24671282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚14740.93479119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70289577 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145484216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75194645 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75204723 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚674 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10336359395840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚338 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚648527646497792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8450490645504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21584324 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2339683030827520 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2331231168565248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10336314427392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46255939 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75194645 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2331231194738688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8451836088832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2330311969314176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70289571 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584¢³‡²J4¿#/ _wandbruntime‚14740  _runtime‚14740zßüa À#† _step‚675 2loss/FBt2/assignment_loss‚0.040128771857086076 _runtime‚14740.970623009 2loss/SRqqt2/assignment_loss‚0.5186219841472389 1loss/SRqqt1/detection_loss‚0.8918088274549406 /loss/FBt2/detection_loss‚0.4745651187549811 %loss/total_loss‚8.041747989418058 % _timestamp‚1.7848539776134233e+09 0loss/FRt1/assignment_loss‚1.6226991446828654 0loss/FRt2/assignment_loss‚0.7483586438067324 1loss/SRqqt1/assignment_loss‚0.596285093521359 /loss/FRt2/detection_loss‚1.4308686853073924 epoch‚28 4loss/FBt1/assignment_loss‚0.00011872823550349396 /loss/FRt1/detection_loss‚0.5633521765981054 1loss/SRqqt2/detection_loss‚0.6074181433046051 /loss/FBt1/detection_loss‚0.5475226717472479 trainer/global_step‚8099>rCÁ#‹ 2loss/SRqqt2/assignment_loss‚0.5186219841472389 1loss/SRqqt1/detection_loss‚0.8918088274549406 /loss/FBt2/detection_loss‚0.4745651187549811 %loss/total_loss‚8.041747989418058 % _timestamp‚1.7848539776134233e+09 0loss/FRt1/assignment_loss‚1.6226991446828654 0loss/FRt2/assignment_loss‚0.7483586438067324 1loss/SRqqt1/assignment_loss‚0.596285093521359 /loss/FRt2/detection_loss‚1.4308686853073924 epoch‚28 4loss/FBt1/assignment_loss‚0.00011872823550349396 /loss/FRt1/detection_loss‚0.5633521765981054 1loss/SRqqt2/detection_loss‚0.6074181433046051 /loss/FBt1/detection_loss‚0.5475226717472479 trainer/global_step‚8099 _step‚675 2loss/FBt2/assignment_loss‚0.040128771857086076 _runtime‚14740.970623009£g–¯“Â#‚(:ˆ ÝëŠÓ£¼³ gpu.0.gpu‚41.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚213.447' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚71.149 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}¤lEÃ#‚(:º  ÝëŠÓÞ¦Û³ disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8576164245605 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 380.67578125 disk.sda2.out‚ 380.67578125# proc.memory.rssMB‚ 2924.87109375 disk.sda2.in‚0.03125 disk.drbd1285.out‚0* proc.memory.percent‚11.901330947875977 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.sda1.in‚0 network.recv‚978411 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 cpu‚25.329233478316688 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 77.01953125 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0# disk.dm-0.out‚43145.90380859375 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0( disk./.usagePercent‚70.5137584389623& disk.nvme0n1.out‚43145.90380859375 disk.drbd2118.in‚0( proc.memory.availableMB‚ 1918.2109375 network.sent‚ 1.4232925e+07 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0$ memory_percent‚92.19477971394858 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0Т!>žÄ#‚(:“ ìëŠÓýÅɳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚277.434' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.47800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©L¾èAÅ#‚(:¶  ìëŠÓûÝæ³ disk.drbd1584.in‚0 cpu‚25.329152825926727 disk.drbd1289.in‚0 network.recv‚981017) disk./.usagePercent‚70.51402297788496 disk.drbd2041.in‚0 disk.drbd1828.out‚0$ memory_percent‚92.19481150309244 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0# disk./.usageGB‚1260.86234664917 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.sda.in‚0.03125 disk.drbd2111.in‚0 disk.rbd1.out‚ 77.01953125 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 380.9609375# proc.memory.rssMB‚ 2924.87109375* proc.memory.percent‚11.901330947875977 network.sent‚ 1.4293009e+07 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0' proc.memory.availableMB‚ 1918.203125 disk.dm-0.in‚ 98.87109375& disk.nvme0n1.out‚43146.57958984375 disk.drbd1550.in‚0 disk.sda2.out‚ 380.9609375 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0# disk.dm-0.out‚43146.57958984375 disk.drbd1552.in‚0¬ÈIžÆ#‚(:“ ûëŠÓßÏ­³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚265.613' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.53766666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åáÈEÇ#‚(:º  ûëŠÓ±Áг network.sent‚ 1.4296035e+07 disk.drbd2041.out‚0# disk.dm-0.out‚43147.40283203125 disk.drbd1510.in‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.51401764443894 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 381.2578125 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 77.18359375 disk.nvme0n1.in‚ 98.87109375 disk.sda.out‚ 381.2578125 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2924.87109375 disk.rbd1.in‚0% disk./.usageGB‚1260.8622512817383 disk.drbd2111.out‚0& disk.nvme0n1.out‚43147.40283203125 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 cpu‚25.329090162718195 disk.drbd1965.in‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1918.20703125* proc.memory.percent‚11.901330947875977 disk.drbd1289.in‚0 network.recv‚981955$ memory_percent‚92.19479560852051 disk.drbd1673.in‚0æ0›SœÈ#‚(:‘ ŠìŠÓÁ…г gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚256.619' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.53966666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'g~åCÉ#‚(:¸  ŠìŠÓç”ï³ proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.sda.out‚ 381.984375# proc.memory.rssMB‚ 2925.52734375 disk.drbd1584.out‚0 disk.drbd1817.in‚0# disk.dm-0.out‚43147.89111328125 cpu‚25.331420959047332 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda2.out‚ 381.984375 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd1828.out‚0$ memory_percent‚92.19784736633301 network.sent‚ 1.4299568e+07 network.recv‚983027 disk.sda1.in‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1917.45703125% disk./.usageGB‚1260.8621406555176 disk.rbd1.out‚ 77.18359375& disk.nvme0n1.out‚43147.89111328125 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1356.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51401145764154 disk.drbd1931.in‚0 disk.drbd1285.out‚0(otÍ4Ê#/ _wandbruntime‚14795  _runtime‚14795%wzðþË#ìu 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6551569104194641 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7449513077735901 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6131130456924438 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3473559319972992 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7075939774513245 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8763022422790527 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.246811181306839 particle/accuracy_2_of_4‚1 trainer/global_step‚8119 4"Purity/0FRt*SRqqt0FBt/event_purity‚ 0.64404296875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9340091943740845 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7630985975265503 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15203356742858887 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8642697930335999 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7045036554336548 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974471926689148 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2566916048526764 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.687813401222229 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7704004645347595 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8149369955062866 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.748126208782196 *jet/accuracy_1_of_3‚0.9365158081054688 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39190635085105896 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8183752298355103 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.735077440738678 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6632228493690491 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8763022422790527 *jet/accuracy_2_of_3‚0.6034353375434875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6206547021865845 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2852305769920349 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.523624837398529 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7257556915283203 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99635249376297 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04262056201696396 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02021120674908161 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.877261221408844 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33565929532051086 +jet/accuracy_4_of_4‚0.02295798622071743 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4544801414012909 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6922324895858765 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3703506588935852 particle/accuracy_2_of_3‚1 'particle/f_score‚0.6526013016700745 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.326698362827301 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7613703608512878 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 (particle/accuracy‚0.8544456362724304 % _timestamp‚1.7848540317620366e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11962640285491943 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39395371079444885 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6904625296592712 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.748845636844635 /particle/accuracy_4_of_4‚0.9685583114624023 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7436590194702148 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925484657287598 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878942370414734 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3339259624481201 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚7.939197328331238 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3356066346168518 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7993687391281128 particle/accuracy_3_of_4‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7284101247787476 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_1_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2833899259567261 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02674945816397667 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8084055185317993 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33565929532051086 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"PuÒ…*ôrity/1FRt0SRqqt1FBt/event_purity‚0.6984310746192932 _step‚676 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6575445532798767 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48933786153793335 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26416558027267456 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8041840195655823 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26861777901649475 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45797300338745117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 epoch‚28 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38606810569763184 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7284457087516785 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 6"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7488973736763 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6869526505470276 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7898662686347961 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13852572441101074 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9898831248283386 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7465271949768066 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.479805052280426 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07707682996988297 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12019497901201248 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3792323172092438 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8145700693130493 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6146093010902405 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8195809721946716 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6100273132324219 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6769536733627319 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24473679065704346 +jet/accuracy_2_of_2‚0.16563200950622559 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6610843539237976 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5933899879455566 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.721981406211853 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9157407879829407 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7660146355628967 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9663068056106567 *jet/accuracy_3_of_4‚0.1148163229227066 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6671423316001892 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2737889885902405 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8443081974983215 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6372308135032654 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34649842977523804 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2081911712884903 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02087497152388096 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8090367913246155 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9137777090072632 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9928030371665955 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6582098603248596 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13318443298339844 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21149389445781708 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.15417297184467316 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6610843539237976 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05079684033989906 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2584035098552704 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.34652018547058105 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45633989572525024 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34885284304618835 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05944131687283516 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3643209934234619 _runtime‚14795.120107857 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27083250880241394 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25479015707969666 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025757255032658577 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6884213089942932 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_3‚0.07208133488893509 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7validation_average_jet_accuracy‚0.40379053354263306&¬ˆ÷:Ì#ñu 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6904625296592712 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.748845636844635 /particle/accuracy_4_of_4‚0.9685583114624023 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7436590194702148 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925484657287598 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878942370414734 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3339259624481201 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚7.939197328331238 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3356066346168518 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7993687391281128 particle/accuracy_3_of_4‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7284101247787476 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_1_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1041666641831398 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2833899259567261 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02674945816397667 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8084055185317993 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33565929532051086 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6984310746192932 _step‚676 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6575445532798767 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48933786153793335 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26416558027267456 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8041840195655823 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26861777901649475 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45797300338745117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 epoch‚28 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38606810569763184 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7284457087516785 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 6"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7488973736763 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6869526505470276 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7898662686347961 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13852572441101074 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9898831248283386 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7465271949768066 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.479805052280426 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07707682996988297 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12019497901201248 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3792323172092438 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8145700693130493 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6146093010902405 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8195809721946716 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6100273132324219 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6769536733627319 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24473679065704346 +jet/accuracy_2_of_2‚0.16563200950622559 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6610843539237976 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5933899879455566 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.721981406211853 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9157407879829407 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7660146355628967 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9663068056106567 *jet/accuracy_3_of_4‚0.1148163229227066 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6671423316001892 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2737889885902405 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8443081974983215 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6372308135032654 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34649842977523804 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2081911712884903 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02087497152388096 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8090367913246155 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9137777090072632 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9928030371665955 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6582098603248596 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13318443298339844 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21149389445781708 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.15417297184467316 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6610843539237976 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05079684033989906 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2584035098552704 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.34652018547058105 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45633989572525024 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34885284304618835 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05944131687283516 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3643209934234619 _runtime‚14795.120107857 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27083250880241394 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25479015707969666 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025757255032658577 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6884213089942932 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_3‚0.07208133488893509 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7validation_average_jet_accuracy‚0.40379053354263306 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6551569104194641 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7449513077735901 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6131130456924438 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3473559319972992 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7075939774513245 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8763022422790527 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.246811181306839 particle/accuracy_2_of_4‚1 trainer/global_step‚8119 4"Purity/0FRt*SRqqt0FBt/event_purity‚ 0.64404296875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9340091943740845 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7630985975265503 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15203356742858887 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8642697930335999 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1041666641831398 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7045036554336548 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974471926689148 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2566916048526764 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.687813401222229 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7704004645347595 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8149369955062866 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.748126208782196 *jet/accuracy_1_of_3‚0.9365158081054688 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39190635085105896 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8183752298355103 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.735077440738678 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6632228493690491 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8763022422790527 *jet/accuracy_2_of_3‚0.6034353375434875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6206547021865845 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2852305769920349 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.523624837398529 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7257556915283203 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99635249376297 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04262056201696396 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02021120674908161 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.877261221408844 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33565929532051086 +jet/accuracy_4_of_4‚0.02295798622071743 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4544801414012909 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6922324895858765 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3703506588935852 particle/accuracy_2_of_3‚1 'particle/f_score‚0.6526013016700745 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.326698362827301 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7613703608512878 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 (particle/accuracy‚0.8544456362724304 % _timestamp‚1.7848540317620366e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11962640285491943 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39395371079444885¤€ h:Í#‚(:¯  ™ìŠÓ¤–ð³) disk./.usagePercent‚70.51422394213104 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 77.2158203125 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0$ memory_percent‚86.83967590332031 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 382.28125 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 382.28125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 cpu‚25.331372325451085 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 network.recv‚984115 disk.dm-0.out‚44118.04296875# disk.nvme0n1.out‚44118.04296875 disk.drbd1931.in‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.904001235961914% disk./.usageGB‚1260.8659400939941 disk.dm-0.in‚ 98.87109375 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0& proc.memory.availableMB‚ 3234.28125 network.sent‚ 1.4302017e+07 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 98.87109375JĬ‰Î#‚(:’ ™ìŠÓÛÿÖ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚277.85' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.61666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉO„%žÏ#‚(:“ ¨ìŠÓ¿­Ø³ gpu.0.gpu‚83.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚289.728' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.57600000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wç„<Ð#‚(:±  ¨ìŠÓøÐô³ disk.nvme0n1.in‚ 98.87109375# proc.memory.rssMB‚ 2925.52734375 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17# memory_percent‚92.0955499013265 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0! disk.nvme0n1.out‚ 45087.765625 disk.sda1.in‚0 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 77.2158203125 disk.drbd1226.out‚0* proc.memory.percent‚11.904001235961914% disk./.usageGB‚1260.8620910644531 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1942.59765625 network.sent‚ 1.4355895e+07) disk./.usagePercent‚70.51400868424962 disk.drbd1550.out‚0 disk.sda.out‚ 382.5859375 disk.drbd1938.in‚0 disk.dm-0.out‚ 45087.765625 network.recv‚986630 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.sda2.out‚ 382.5859375 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 cpu‚25.331390999459295 disk.drbd1673.out‚0 disk.drbd1931.in‚0šÙ÷üžÑ#‚(:“ ·ìŠÓÀºº³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚295.954' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.65133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¡ ¡HÒ#‚(:½  ·ìŠÓßåÛ³ network.recv‚987709) disk./.usagePercent‚70.51400868424962 disk.sda2.out‚ 382.89453125 disk.rbd1.out‚ 77.3798828125 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.904001235961914 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125# disk.dm-0.out‚45088.39794921875 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 382.89453125 disk.drbd1226.in‚0) proc.memory.availableMB‚ 1942.59765625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚0.03125 disk.dm-0.in‚ 98.87109375 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0# memory_percent‚92.0955499013265 network.sent‚ 1.4358978e+07 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 cpu‚25.331326400767757 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1289.out‚0& disk.nvme0n1.out‚45088.39794921875 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8620910644531 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 98.87109375S]ôžÓ#‚(:“ ÆìŠÓåèÚ³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.25' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.08333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆøFÔ#‚(:»  ÆìŠÓÚøõ³ disk.drbd2111.in‚0& disk.nvme0n1.out‚45088.78857421875# memory_percent‚92.0955499013265 network.sent‚ 1.4362506e+07 disk.drbd1552.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 disk.drbd1938.in‚0) disk./.usagePercent‚70.51401081762802% disk./.usageGB‚1260.8621292114258 disk.sda.out‚ 383.20703125 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 98.87109375# disk.dm-0.out‚45088.78857421875 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 383.20703125 disk.drbd1550.in‚0 cpu‚25.3312773011284 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 77.3798828125 disk.nvme0n1.in‚ 98.87109375) proc.memory.availableMB‚ 1942.59765625 disk.drbd1673.in‚0 network.recv‚988642 disk.sda.in‚0.03125* proc.memory.percent‚11.904001235961914 disk.drbd2118.out‚0/›v4Õ#/ _wandbruntime‚14851  _runtime‚14851¸@0yÈÖ# trainer/global_step‚8149 % _timestamp‚1.7848540876761048e+09 _runtime‚14851.033341238 _step‚677 & lr-AdamW/pg1‚4.515383079004391e-05 & lr-AdamW/pg2‚4.515383079004391e-05ô7ÛÍ×#Ç & lr-AdamW/pg1‚4.515383079004391e-05 & lr-AdamW/pg2‚4.515383079004391e-05 trainer/global_step‚8149 % _timestamp‚1.7848540876761048e+09 _runtime‚14851.033341238 _step‚677¥¯)¶4Ø#/ _wandbruntime‚14851  _runtime‚14851BÛÔ84Ù#„P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚324 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146477813 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642091063515136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8507621569024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24844667 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8507621569024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =6DeviceSta_áxÖ!tsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652496627220992 % _timestamp‚1.7848540876771798e+09 _runtime‚14851.034768477 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10405195614720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8492032785864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2355966048266752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652496212057088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75688827 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24844343 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2355964676657152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146487883 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46564914 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2355025660205486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8493377397150 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2346532257380368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚8149 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2347458400524288 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10405148541952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2347458426697728 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2347458426697728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚642091431606272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2355024290166232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8506276132864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2355964676657152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75698891 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70788992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21720576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70788986 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8507621569024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24844667 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8507621569024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652496627220992 % _timestamp‚1.7848540876771798e+09 _runtime‚14851.034768477 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10405195614720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8492032785864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2355966048266752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652496212057088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75688827 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24844343 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2355964676657152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146487883 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46564914 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2355025660205486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8493377397150 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2346532257380368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚8149 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2347458400524288 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10405148541952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2347458426697728 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2347458426697728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚642091431606272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2355024290166232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8506276132864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2355964676657152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75698891 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70788992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21720576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70788986¦è6Ñ4Û#/ _wandbruntime‚14852  _runtime‚14852zÓãÎ'Ü#ŒN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2356245980072448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21723216 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70797353 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21723211 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146505346 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2356245980072448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8494398198902 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146495262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8508644084224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8493053587504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2346811084582112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2347737335988224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70797359 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚321 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2347737309814784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75697909 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46570560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8507298640896 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70797359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75707987 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2347737335988224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146495262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚316 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2356244608455680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2346811059154144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75707987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2347737309814784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146505346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70797353 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚679 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642169649881088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚8149 # _timestamp‚1.78485408915951e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75697909 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24847665 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10406450151936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642170017972224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10406405183488 _runtime‚14852.517201149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46570881 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8508644084224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652576468124160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2346811084582112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2347737335988224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70797359 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚321 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2347737309814784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75697909 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46570560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8507298640896 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70797359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75707987 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2347737335988224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146495262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚316 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_enÝðged/allocated_bytes.all.freed‚2356244608455680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2346811059154144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75707987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2347737309814784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146505346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70797353 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _step‚679 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642169649881088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚8149 # _timestamp‚1.78485408915951e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75697909 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24847665 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10406450151936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642170017972224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10406405183488 _runtime‚14852.517201149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46570881 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8508644084224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652576468124160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146505346 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2356245980072448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8494398198902 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146495262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8508644084224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8493053587504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0§_Ù…4Þ#/ _wandbruntime‚14852  _runtime‚14852¯1ß#ˆ 2loss/SRqqt1/assignment_loss‚0.5173594728778246 2loss/FBt2/assignment_loss‚0.017031254578148004 1loss/SRqqt2/detection_loss‚0.6054161165773698 0loss/FBt1/detection_loss‚0.32149961723112747 %loss/total_loss‚7.406502027091745 _step‚680 0loss/FRt2/assignment_loss‚0.6824936301382862 2loss/SRqqt2/assignment_loss‚0.4750029970973375 4loss/FBt1/assignment_loss‚3.3191553720893364e-05 /loss/FRt1/detection_loss‚0.5263205867301339 trainer/global_step‚8149 % _timestamp‚1.7848540891913002e+09 _runtime‚14852.548539138 /loss/FRt2/detection_loss‚1.3305993930710651 1loss/SRqqt1/detection_loss‚0.8324087625545309 0loss/FBt2/detection_loss‚0.46565098967403173 epoch‚29 /loss/FRt1/assignment_loss‚1.632686015008169Ây2fà# 4loss/FBt1/assignment_loss‚3.3191553720893364e-05 /loss/FRt1/detection_loss‚0.5263205867301339 trainer/global_step‚8149 % _timestamp‚1.7848540891913002e+09 _runtime‚14852.548539138 /loss/FRt2/detection_loss‚1.3305993930710651 1loss/SRqqt1/detection_loss‚0.8324087625545309 0loss/FBt2/detection_loss‚0.46565098967403173 epoch‚29 /loss/FRt1/assignment_loss‚1.632686015008169 2loss/SRqqt1/assignment_loss‚0.5173594728778246 2loss/FBt2/assignment_loss‚0.017031254578148004 1loss/SRqqt2/detection_loss‚0.6054161165773698 0loss/FBt1/detection_loss‚0.32149961723112747 %loss/total_loss‚7.406502027091745 _step‚680 0loss/FRt2/assignment_loss‚0.6824936301382862 2loss/SRqqt2/assignment_loss‚0.4750029970973375¨Ÿ]ë%“á#‚(:ˆ ÕìŠÓËó¾³ gpu.0.gpu‚61.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚226.812' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚75.604 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0."CÁEâ#‚(:º  ÕìŠÓóÆܳ disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 77.412109375 cpu‚25.33121295742004) disk./.usagePercent‚70.51400975093883 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 383.6640625 disk.drbd2078.out‚0 disk.drbd2041.in‚0 network.sent‚ 1.4365004e+07) proc.memory.availableMB‚ 1942.53515625 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.sda2.in‚0.03125 disk.drbd1828.in‚0# disk.dm-0.out‚45089.20361328125 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2925.52734375 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0% disk./.usageGB‚1260.8621101379395 disk.sda2.out‚ 383.6640625 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125$ memory_percent‚92.09580421447754 network.recv‚989585 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0& disk.nvme0n1.out‚45089.20361328125 disk.drbd1550.out‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.904001235961914:ª[ã#‚(:’ äìŠÓù„Ú³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚263.96' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.98666666666665 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0._à¤9ä#‚(:®  äìŠÓ¢Áø³ disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.09580421447754 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚384.25# disk.dm-0.out‚45089.37158203125 disk.rbd1.out‚ 77.412109375 disk.nvme0n1.in‚ 98.87109375 disk.drbd1289.out‚0* proc.memory.percent‚11.904001235961914 network.sent‚ 1.4425053e+07$ disk./.usageGB‚1260.862060546875# proc.memory.rssMB‚ 2925.52734375( disk./.usagePercent‚70.5140069775469 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0& disk.nvme0n1.out‚45089.37158203125 cpu‚25.33118078358691 network.recv‚992331 disk.drbd1584.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1942.53515625 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.sda.out‚384.25 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd2041.in‚0 disk.drbd1285.in‚0]FÊÝå#‚(:’ óìŠÓ€»Â³ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.05' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.68333333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PzÖFæ#‚(:»  óìŠÓÌòܳ disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 384.68359375# disk.dm-0.out‚45089.77978515625 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1828.out‚0 network.recv‚993313 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd2041.out‚0 disk.rbd1.out‚ 77.755859375 disk.sda1.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1965.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 384.68359375 disk.drbd1931.out‚0( disk./.usagePercent‚70.5140069775469 disk.sda.in‚0.03125 disk.drbd1289.out‚0$ memory_percent‚92.09580421447754$ disk./.usageGB‚1260.862060546875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2925.52734375* proc.memory.percent‚11.904001235961914 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0) proc.memory.availableMB‚ 1942.53515625 cpu‚25.331133478168745 disk.drbd1226.in‚0 network.sent‚ 1.4428036e+07 disk.drbd1285.in‚0& disk.nvme0n1.out‚45089.77978515625 disk.drbd1510.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1931.in‚05éÇ´Cç#‚(:¸  ‚íŠÓ²Ü³ disk.nvme0n1.in‚ 98.87109375 disk.drbd1931.out‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51400825757393# disk.dm-0.out‚45089.95947265625 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.sda.out‚ 385.015625 disk.dm-0.in‚ 98.87109375 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2925.52734375 cpu‚25.33108553281923 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0& disk.nvme0n1.out‚45089.95947265625$ memory_percent‚92.09580421447754* proc.memory.percent‚11.904001235961914 proc.cpu.threads‚17 network.sent‚ 1.4431053e+07 network.recv‚994459 disk.rbd1.out‚ 77.755859375 disk.drbd1584.in‚0% disk./.usageGB‚1260.8620834350586 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1828.out‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 385.015625 disk.drbd1550.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1942.53515625 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71207336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646020381812736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2360472371497136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46844523 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10467739548160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8543136668672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2369961330738176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8557465689088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8558811125248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784854174395799e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2369962702347776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2361403891222528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2369016852777094 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚8199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8558811125248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656488074288128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚646020749903872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24992307 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656488489452032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2361403891222528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76154843 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147362185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚14937.753296332 _step‚682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8557465689088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2361403865049088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71207342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2369962702347776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2360472346069168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10467692475392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2369961330738176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76144779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21852571 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2369015482737840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005{Ô _ð#ˆP TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21852571 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2369015482737840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46844883 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71207336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646020381812736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2360472371497136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46844523 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10467739548160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8543136668672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2369961330738176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8557465689088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8558811125248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784854174395799e+09 WBDeviceStatsMoë¾U¨¯nitor.on_train_batch_start/active_bytes.all.allocated‚2369962702347776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2361403891222528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2369016852777094 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚8199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8558811125248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656488074288128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚646020749903872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24992307 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656488489452032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2361403891222528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76154843 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147362185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚14937.753296332 _step‚682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8557465689088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2361403865049088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71207342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2369962702347776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2360472346069168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10467692475392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2369961330738176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76144779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ª¶¦´%4ñ#/ _wandbruntime‚14939  _runtime‚14939Õ£é'ò#N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76163939 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147369564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656567906973696 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76153861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚646099336269824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2369296675352622 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2360751173270912 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2369295305313256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656568328421888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46850118 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24994907 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2361682800513024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46850473 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2361682774339584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24995257 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147379648 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71215703 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76163939 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21855216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2361682800513024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8545502081710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71215709 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8559833640448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8559833640448 _step‚683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147379648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71215703 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76153861 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8558488197120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10468992152064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2361682800513024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46850473 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2361682774339584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24995257 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147379648 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71215703 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76163939 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21855216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2361682800513024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8545502081710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71215709 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8559833640448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8559833640448 _step‚683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147379648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71215703 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76153861 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8558488197120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10468992152064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147369564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656567906973696 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76153861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚646099336269824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2369296675352622 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2360751173270912 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2369295305313256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656568328421888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46850118 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24994907«ë—“q4ô#/ _wandbruntime‚14939  _runtime‚14939 Ëè¼ õ#… 0loss/FRt1/assignment_loss‚1.6431202275014063 0loss/FRt2/assignment_loss‚0.6719056166507102 2loss/SRqqt1/assignment_loss‚0.4540153808009545 3loss/SRqqt2/assignment_loss‚0.46652233833240103 /loss/FRt2/detection_loss‚1.3851243309867527 $loss/total_loss‚7.98617636152663 _step‚684 1loss/SRqqt2/detection_loss‚0.6079289220786186 /loss/FBt2/detection_loss‚0.5069360312940482 _runtime‚14939.266506897 3loss/FBt1/assignment_loss‚9.740189851882977e-05 1loss/SRqqt1/detection_loss‚0.8211553974542767 .loss/FBt1/detection_loss‚0.843150025534053 trainer/global_step‚8199 % _timestamp‚1.7848541759092844e+09 2loss/FBt2/assignment_loss‚0.038753880320700894 /loss/FRt1/detection_loss‚0.5474668086741895 epoch‚29G ö#Š _step‚684 1loss/SRqqt2/detection_loss‚0.6079289220786186 /loss/FBt2/detection_loss‚0.5069360312940482 _runtime‚14939.266506897 3loss/FBt1/assignment_loss‚9.740189851882977e-05 1loss/SRqqt1/detection_loss‚0.8211553974542767 .loss/FBt1/detection_loss‚0.843150025534053 trainer/global_step‚8199 % _timestamp‚1.7848541759092844e+09 2loss/FBt2/assignment_loss‚0.038753880320700894 /loss/FRt1/detection_loss‚0.5474668086741895 epoch‚29 0loss/FRt1/assignment_loss‚1.6431202275014063 0loss/FRt2/assignment_loss‚0.6719056166507102 2loss/SRqqt1/assignment_loss‚0.4540153808009545 3loss/SRqqt2/assignment_loss‚0.46652233833240103 /loss/FRt2/detection_loss‚1.3851243309867527 $loss/total_loss‚7.98617636152663¬f²ë>÷#‚(:³   íŠÓ»°û³ disk.drbd2078.out‚0 disk.nvme0n1.in‚ 98.87109375* proc.memory.percent‚11.904001235961914 disk.drbd2118.out‚0& disk.nvme0n1.out‚45091.44384765625 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.sda.out‚ 386.0546875 disk.drbd1550.out‚0% disk./.usageGB‚1260.8630104064941 disk.sda1.in‚0# disk.dm-0.out‚45091.44384765625 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1510.out‚0 cpu‚25.330938894330696 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0 network.recv‚996452 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 network.sent‚ 1.4437581e+07) disk./.usagePercent‚70.51406009866926 disk.drbd2118.in‚0" proc.memory.availableMB‚1942.5 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 77.7880859375 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0" memory_percent‚92.095947265625 disk.drbd1289.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 386.0546875 disk.drbd1285.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚07Û†œø#‚(:‘  íŠÓÖ¥Þ³ gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚294.113' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.03766666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IWò;>ù#‚(:³  ¯íŠÓûă´# disk.nvme0n1.out‚45092.59765625 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1942.50390625 disk.drbd2118.out‚0 cpu‚25.33085766562471 disk.sda.in‚0.03125 disk.nvme0n1.in‚ 98.87109375 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2925.52734375* proc.memory.percent‚11.904001235961914% disk./.usageGB‚1260.8630104064941 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 network.sent‚ 1.4497056e+07 disk.sda2.in‚0.03125 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 78.0458984375 disk.drbd1226.in‚0$ memory_percent‚92.09593137105307 proc.cpu.threads‚17 disk.sda2.out‚ 386.359375 disk.drbd1965.out‚0 disk.drbd1550.out‚0 network.recv‚999163 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.sda.out‚ 386.359375 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.dm-0.out‚45092.59765625 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd2078.out‚0) disk./.usagePercent‚70.51406009866926 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0áÍœ™ú#‚(:’ ¯íŠÓ„Ûã³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.27' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.08999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôŸUžû#‚(:“ ¾íŠÓ¹¹¹³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚260.899' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.96633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Œï(õü#‚(:¿  ¾íŠÓöéس) proc.memory.availableMB‚ 1942.50390625 network.recv‚ 1.000103e+06( disk./.usagePercent‚70.5137656924489 disk.drbd1938.in‚0" disk.dm-0.out‚45092.9365234375 disk.drbd1828.in‚0 disk.sda1.out‚%NÁxU0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.rbd1.out‚ 78.0458984375 disk.drbd1510.out‚0 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.904001235961914 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0$ memory_percent‚92.09593137105307 disk.sda2.out‚ 386.8203125 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 386.8203125 disk.dm-0.in‚ 98.87109375 disk.nvme0n1.in‚ 98.87109375 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0% disk.nvme0n1.out‚45092.9365234375 disk.sda.in‚0.03125 network.sent‚ 1.4500037e+07 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 cpu‚25.330794668235836% disk./.usageGB‚1260.8577461242676 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 úøžý#‚(:“ ÍíŠÓ‚Èá³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚111.977' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.32566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àæØêLþ#‚(:Á  ÍíŠÓïÀÿ³ network.recv‚ 1.001067e+06 disk.drbd1931.out‚0 cpu‚25.330746198092488 network.sent‚ 1.4503062e+07 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd2078.in‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51382030693615 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd1817.out‚0& proc.memory.availableMB‚ 1942.46875 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0$ memory_percent‚92.09607442220053 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.sda2.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.sda.out‚ 387.14453125 disk.drbd1965.in‚0& disk.nvme0n1.out‚45093.68212890625 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0# disk.dm-0.out‚45093.68212890625 proc.cpu.threads‚17 disk.rbd1.out‚ 78.2021484375 disk.drbd1226.out‚0 disk.sda.in‚0.03125 disk.drbd1828.in‚0 disk.sda2.out‚ 387.14453125% disk./.usageGB‚1260.8587226867676 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 %žÿ#‚(:“ ÜíŠÓËŒ¯³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.987' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.99566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª9RïF€$‚(:»  ÜíŠÓ¾Ù̳ disk.rbd1.out‚ 78.3818359375 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0# proc.memory.rssMB‚ 2925.52734375) disk./.usagePercent‚70.51381817355774 disk.drbd2078.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd1938.out‚0 disk.sda2.out‚ 387.390625& proc.memory.availableMB‚ 1942.46875 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd2111.in‚0 network.sent‚ 1.4506061e+07 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0$ memory_percent‚92.09607442220053 proc.cpu.threads‚17 network.recv‚ 1.002076e+06 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 387.390625 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 cpu‚25.33066703542176 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚45093.85009765625$ disk./.usageGB‚1260.858684539795 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 98.87109375& disk.nvme0n1.out‚45093.85009765625 disk.drbd2041.in‚0³1ž$‚(:“ ëíŠÓîíÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚202.174' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.39133333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T êM‚$‚(:  ëíŠÓÈø³ disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 98.87109375# disk.dm-0.out‚45094.19873046875) proc.memory.availableMB‚ 1942.71484375 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 387.66796875 disk.drbd1356.out‚0& disk.nvme0n1.out‚45094.19873046875 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0$ disk./.usageGB‚1260.858684539795 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 cpu‚25.330602035003956* proc.memory.percent‚11.904001235961914 disk.drbd1289.in‚0 disk.drbd2118.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.sda.out‚ 387.66796875 disk.drbd1550.in‚0 disk.drbd1931.in‚0 network.sent‚ 1.4509747e+07 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0# memory_percent‚92.0950730641683 disk.drbd1510.out‚0 disk.sda1.in‚0 network.recv‚ 1.003331e+06) disk./.usagePercent‚70.51381817355774 disk.rbd1.out‚ 78.5458984375 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2041.out‚054Ð\4ƒ$/ _wandbruntime‚15024  _runtime‚15024ˆ¢‚©Ê„$Ä trainer/global_step‚8249 % _timestamp‚1.7848542615594347e+09 _runtime‚15024.916695044 _step‚685 ' lr-AdamW/pg1‚4.3852087850305846e-05 ' lr-AdamW/pg2‚4.3852087850305846e-05 4YpÏ…$É ' lr-AdamW/pg1‚4.3852087850305846e-05 ' lr-AdamW/pg2‚4.3852087850305846e-05 trainer/global_step‚8249 % _timestamp‚1.7848542615594347e+09 _runtime‚15024.916695044 _step‚685­h÷4†$/ _wandbruntime‚15024  _runtime‚15024$Ž Î (‡$„P WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8608655245312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚355 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47123506 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76600731 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚15024.920916556 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8594240551480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8610000681472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25138935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71625686 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8608655245312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76610795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2374412434757968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2375349329573888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2374412460185936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148226417 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2383957984819200 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71625686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚350 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649949700110336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10530337549824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2375349329573888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47123861 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71625692 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148236487 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660480405751296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21984576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2383957984819200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76600731 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595585162766 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8610000681472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649950068201472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76610795 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 % _timestamp‚1.7848542615605626e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2383959356428800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚660479984295936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21984571 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0„ÑP(ˆ$‰P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47123506 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76600731 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚15024.920916556 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8594240551480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8610000681472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25138935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71625686 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8608655245312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76610795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2374412434757968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2375349329573888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2374412460185936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148226417 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2383957984819200 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71625686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚350 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649949700110336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10530337549824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2375349329573888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47123861 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71625692 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148236487 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660480405751296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21984576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2383957984819200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76600731 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595585162766 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8610000681472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649950068201472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76610795 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 % _timestamp‚1.7848542615605626e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2383959356428800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚660479984295936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21984571 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8608655245312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚355 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚686®Þ6Tþ4‰$/ _wandbruntime‚15026  _runtime‚15026|v¾ ;Š$ŽN E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2383286497884864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25142147 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148253950 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8609677753344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76609813 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76609813 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71634059 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71634053 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 % _timestamp‚1.7848542630392506e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76619891 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148243866 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 475åÎû‡Ù15172864 _step‚687 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21987211 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8611023196672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47129010 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚15026.396883573 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2375628238864384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47129363 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148253950 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25141799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8596605964518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2384239288234496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660560244327936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2374691236531744 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10531532209152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚8249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148243866 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71634053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2375628265037824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8611023196672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚6500282864762880:¹'‹$“N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71634053 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 % _timestamp‚1.7848542630392506e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76619891 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148243866 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _step‚687 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21987211 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8611023196672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47129010 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚15026.396883573 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2375628238864384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47129363 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148253950 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25141799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8596605964518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2384239288234496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660560244327936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2374691236531744 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10531532209152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚8249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148243866 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71634053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2375628265037824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8611023196672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚650028286476288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2383286497884864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25142147 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148253950 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8609677753344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76609813 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76609813 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71634059 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005¯ƒsª4Œ$/ _wandbruntime‚15026  _runtime‚15026ªË Ú $ƒ /loss/FRt1/detection_loss‚0.5256941623075134 %loss/total_loss‚7.997226778672713 trainer/global_step‚8249 0loss/FRt1/assignment_loss‚1.6684054126303027 2loss/FBt1/assignment_loss‚0.008648825044213382 /loss/FBt1/detection_loss‚0.5817271887572133 /loss/FBt2/detection_loss‚0.4162597342651817 $ _timestamp‚1.784854263072779e+09 2loss/SRqqt1/assignment_loss‚0.5833522777833262 /loss/FRt2/detection_loss‚1.5150077361826628 epoch‚29 _runtime‚15026.430071722 _step‚688 0loss/FRt2/assignment_loss‚0.6946542940914494 1loss/SRqqt1/detection_loss‚0.8409267831662232 1loss/SRqqt2/detection_loss‚0.6478560800062949 1loss/SRqqt2/assignment_loss‚0.481486712551374 2loss/FBt2/assignment_loss‚0.033207571886957846T[IŽ$ˆ _step‚688 0loss/FRt2/assignment_loss‚0.6946542940914494 1loss/SRqqt1/detection_loss‚0.8409267831662232 1loss/SRqqt2/detection_loss‚0.6478560800062949 1loss/SRqqt2/assignment_loss‚0.481486712551374 2loss/FBt2/assignment_loss‚0.033207571886957846 /loss/FRt1/detection_loss‚0.5256941623075134 %loss/total_loss‚7.997226778672713 trainer/global_step‚8249 0loss/FRt1/assignment_loss‚1.6684054126303027 2loss/FBt1/assignment_loss‚0.008648825044213382 /loss/FBt1/detection_loss‚0.5817271887572133 /loss/FBt2/detection_loss‚0.4162597342651817 $ _timestamp‚1.784854263072779e+09 2loss/SRqqt1/assignment_loss‚0.5833522777833262 /loss/FRt2/detection_loss‚1.5150077361826628 epoch‚29 _runtime‚15026.430071722°ö§U”$‚(:‰ úíŠÓŽÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚305.124' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ŽÊ8N$‚(:à  úíŠÓ‚»‚´ disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.rbd1.in‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1285.in‚0* proc.memory.percent‚11.904001235961914 network.sent‚ 1.4512201e+07 network.recv‚ 1.004352e+06 disk.drbd1356.out‚0 disk.rbd1.out‚ 78.5458984375 disk.drbd1510.out‚0% disk./.usageGB‚1260.8587112426758 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 388.30078125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.51381966692263 disk.drbd1550.in‚0 cpu‚25.330538356855474 disk.drbd1817.in‚0 disk.dm-0.in‚ 98.87109375# disk.dm-0.out‚45094.48095703125 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.09521611531576( proc.memory.availableMB‚ 1942.6796875 disk.drbd2118.out‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚45094.48095703125 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.sda1.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.sda.out‚ 388.30078125 disk.drbd1673.out‚0¿öž‘$‚(:“ ‰îŠÓ×üé³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.653' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.88433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¯˜¶B’$‚(:·  ‰îŠÓé disk.drbd1828.in‚0 disk.drbd1356.out‚0 network.recv‚ 1.007225e+06 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 cpu‚25.330491010136647 disk.drbd2111.in‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 disk.drbd2078.out‚0$ memory_percent‚92.09620157877603& disk.nvme0n1.out‚45095.00244140625 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0$ disk./.usageGB‚1260.858730316162 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1226.out‚0 disk.drbd1289.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.sda1.out‚0 disk.sda.out‚ 388.796875 disk.drbd2041.in‚0 network.sent‚ 1.4572293e+07 disk.drbd1817.out‚0 disk.sda2.out‚ 388.796875 disk.rbd1.out‚ 78.578125 disk.dm-0.in‚ 98.87109375# disk.dm-0.out‚45095.00244140625 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.904001235961914 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51382073361184 disk.drbd1510.out‚0% proc.memory.availableMB‚ 1942.4375`»zTŸ“$‚(:” ˜îŠÓÏä±³ gpu.0.gpu‚42.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚303.177' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.05900000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼w9œ@”$‚(:µ  ˜îŠÓ¶ÙÒ³ disk.drbd1285.in‚0& disk.nvme0n1.out‚45095.20947265625 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1828.out‚0# disk.dm-0.out‚45095.20947265625 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0$ memory_percent‚92.09518432617188% proc.memory.availableMB‚ 1942.6875* proc.memory.percent‚11.904001235961914 network.sent‚ 1.4575287e+07 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 389.09375) disk./.usagePercent‚70.51381753354423 network.recv‚ 1.00834e+06 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2925.52734375 cpu‚25.330429021648115 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.rbd1.out‚ 78.578125 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8586730957031 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 389.09375 disk.drbd1965.in‚0ÍÆÓž•$‚(:“ §îŠÓí…¾³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.687' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.56233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÏ“²<–$‚(:±  §îŠÓëÖ³ disk.drbd2041.out‚0& disk.nvme0n1.out‚45095.54541015625 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1550.out‚0 disk.sda.out‚389.375 disk.sda2.in‚0.03125# proc.memory.rssMB‚ 2925.52734375 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0$ memory_percent‚92.09620157877603 cpu‚25.33034907258374 disk.drbd1931.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd2041.in‚0 disk.rbd1.out‚ 78.828125 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8586730957031% proc.memory.availableMB‚ 1942.4375 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 network.sent‚ 1.4578858e+07# disk.dm-0.out‚45095.54541015625 disk.drbd1931.out‚0 disk.drbd1289.in‚0 network.recv‚ 1.009518e+06 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚389.375) disk./.usagePercent‚70.51381753354423 disk.dm-0.in‚ 98.87109375ˆ»ž—$‚(:“ ¶îŠÓ¿ùÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.397' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.13233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ}WÎB˜$‚(:·  ¶îŠÓƱô³$ memory_percent‚92.09620157877603 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51381860023342 disk.sda.in‚0.03125 disk.sda.out‚ 389.828125 disk.drbd1285.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚45095.70166015625 network.recv‚ 1.010688e+06 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 98.87109375% proc.memory.availableMB‚ 1942.4375% disk./.usageGB‚1260.8586921691895 disk.drbd1584.out‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.904001235961914 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1550.out‚0# disk.dm-0.out‚45095.70166015625 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 network.sent‚ 1.4581848e+07 disk.rbd1.out‚ 78.828125 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1285.out‚0 cpu‚25.33030161667608 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 389.828125 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0ϺŠœ™$‚(:‘ ÅîŠÓí½Å³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚108.026' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.00866666666667 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª‹å1Bš$‚(:·  ÅîŠÓ˜ë³ disk.drbd1226.out‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0" disk.dm-0.out‚45095.9990234375 disk.sda2.in‚0.03125) disk./.usagePercent‚70.51382180030103 disk.drbd2041.in‚0% disk.nvme0n1.out‚45095.9990234375# proc.memory.rssMB‚ 2925.52734375 cpu‚25.33023881364597 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1931.out‚0 network.sent‚ 1.4584334e+07 disk.drbd1226.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.09518432617188* proc.memory.percent‚11.904001235961914 disk.drbd1550.out‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1965.out‚0 network.recv‚ 1.011634e+06 disk.sda2.out‚ 390.15625 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 78.8603515625 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.8587493896484 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 390.15625% proc.memory.availableMB‚ 1942.6875Ù»ËÛ4›$/ _wandbruntime‚15111  _runtime‚15111ÊuëöÈœ$ _step‚689 & lr-AdamW/pg1‚4.320406647616767e-05 & lr-AdamW/pg2‚4.320406647616767e-05 trainer/global_step‚8299 % _timestamp‚1.7848543485748014e+09 _runtime‚15111.932043133#: Í$Ç & lr-AdamW/pg1‚4.320406647616767e-05 & lr-AdamW/pg2‚4.320406647616767e-05 trainer/global_step‚8299 % _timestamp‚1.7848543485748014e+09 _runtime‚15111.932043133 _step‚689±„Šì4ž$/ _wandbruntime‚15111  _runtime‚15111¸ˆ £ó Ÿ$P PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77066747 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2389294794098688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚690 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 # _timestamp‚1.78485434857586e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10592846442496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8645344434288 trainer/global_step‚8299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77056683 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25285972 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2396997867881056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47402184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25285613 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72044042 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2389294794098688 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8659844801536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22116571 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2396999237920310 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2388352548874736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149100719 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77056683 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2397954638900224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8659844801536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10592795175424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8661190237696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77066747 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2397956010509824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664472232941568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8646689045574 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149110789 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2389294820272128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2388352523446768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚15111.93692557©¿m ( $†P PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77066747 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2389294794098688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚690 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 # _timestamp‚1.78485434857586e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10592846442496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8645344434288 trainer/global_step‚8299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77056683 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25285972 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2396997867881056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47402184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25285613 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72044042 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2389294794098688 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8659844801536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22116571 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2396999237920310 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2388352548874736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149100719 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77056683 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2397954638900224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8659844801536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10592795175424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8661190237696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77066747 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2397956010509824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664472232941568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8646689045574 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149110789 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2389294820272128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2388352523446768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚15111.93692557²?»SG4¡$/ _wandbruntime‚15113  _runtime‚15113¶qG'¢$ŽN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149118168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2389573729562624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10594101210112 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22119216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2397279060495838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚15113.409381543 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2397277690456472 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2389573729562624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72052409 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2398235942315520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848543500517602e+09 trainer/global_step‚8299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25288561 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2398234570698752 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2389573703389184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22119211 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚664552074075136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653957972865024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77075843 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2398235942315520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47407772 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2388631325220544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72052403 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77065765 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8660867309568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚363 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8662212752896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149118168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77065765 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2388631350648512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149128252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25288919 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 _step‚691 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8660867309568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72052403 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚653957604773888 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72052409 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47408135 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10594043658752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8646365235928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2389573703389184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Ö0Thu£$“N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2398235942315520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848543500517602e+09 trainer/global_step‚8299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25288561 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2398234570698752 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2389573703389184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22119211 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚664552074075136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653957972865024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77075843 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2398235942315520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47407772 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2388631325220544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72052403 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77065765 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8660867309568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚363 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8662212752896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149118168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77065765 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2388631350648512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149128252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25288919 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/ÌÏN¤reserved_bytes.large_pool.peak‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 _step‚691 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8660867309568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72052403 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚653957604773888 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72052409 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47408135 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10594043658752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8646365235928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2389573703389184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149118168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2389573729562624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10594101210112 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22119216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2397279060495838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _runtime‚15113.409381543 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2397277690456472 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2389573729562624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72052409 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0³bŠt4¤$/ _wandbruntime‚15113  _runtime‚15113òqEÌ ¥$… 0loss/FRt1/assignment_loss‚1.6352036059649955 trainer/global_step‚8299 % _timestamp‚1.7848543500868769e+09 _step‚692 epoch‚29 2loss/SRqqt1/assignment_loss‚0.5885299587927535 2loss/SRqqt2/assignment_loss‚0.4417124906563963 1loss/SRqqt1/detection_loss‚0.7400862686650241 _runtime‚15113.444085633 1loss/SRqqt2/detection_loss‚0.6062391443980438 /loss/FBt1/detection_loss‚0.8432773651937688 /loss/FBt2/detection_loss‚0.4180071121046698 %loss/total_loss‚7.872024203091912 0loss/FRt2/assignment_loss‚0.6668571479487837 3loss/FBt1/assignment_loss‚4.202355957109002e-05 2loss/FBt2/assignment_loss‚0.050626385744042886 .loss/FRt1/detection_loss‚0.550322250299135 /loss/FRt2/detection_loss‚1.3311204497647275ωÆô¦$Š _runtime‚15113.444085633 1loss/SRqqt2/detection_loss‚0.6062391443980438 /loss/FBt1/detection_loss‚0.8432773651937688 /loss/FBt2/detection_loss‚0.4180071121046698 %loss/total_loss‚7.872024203091912 0loss/FRt2/assignment_loss‚0.6668571479487837 3loss/FBt1/assignment_loss‚4.202355957109002e-05 2loss/FBt2/assignment_loss‚0.050626385744042886 .loss/FRt1/detection_loss‚0.550322250299135 /loss/FRt2/detection_loss‚1.3311204497647275 0loss/FRt1/assignment_loss‚1.6352036059649955 trainer/global_step‚8299 % _timestamp‚1.7848543500868769e+09 _step‚692 epoch‚29 2loss/SRqqt1/assignment_loss‚0.5885299587927535 2loss/SRqqt2/assignment_loss‚0.4417124906563963 1loss/SRqqt1/detection_loss‚0.7400862686650241´!:+¨I§$‚(:¾  ÔîŠÓÉíø³ disk.drbd1356.in‚0 network.recv‚ 1.012571e+06 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.sda2.in‚0.03125 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda.in‚0.03125 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 390.6796875) disk./.usagePercent‚70.51381604017934" disk.dm-0.out‚45096.7919921875 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 390.6796875 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 98.87109375% disk.nvme0n1.out‚45096.7919921875 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 78.8603515625* proc.memory.percent‚11.904001235961914 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1942.21484375 cpu‚25.3301755097743 network.sent‚ 1.4587325e+07 disk.drbd1584.in‚0$ memory_percent‚92.09710756937662% disk./.usageGB‚1260.8586463928223 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0^Aˆxž¨$‚(:“ ÔîŠÓòÈÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚259.527' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.50899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"è)L“©$‚(:ˆ ãîŠÓöðÁ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚233.952' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚77.984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07 Ú!Hª$‚(:½  ãîŠÓšœß³ disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 390.9609375 disk.drbd1289.in‚0 network.recv‚ 1.015005e+06 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.out‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51381604017934 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 79.0244140625 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2925.52734375 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1942.21484375 cpu‚25.33011340793083 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 98.87109375* proc.memory.percent‚11.904001235961914 disk.drbd1510.out‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8586463928223 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2118.in‚0 network.sent‚ 1.4647401e+07$ disk.nvme0n1.out‚45097.205078125 disk.drbd2118.out‚0$ memory_percent‚92.09710756937662 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0! disk.dm-0.out‚45097.205078125 disk.sda.out‚ 390.9609375 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0{žphŸ«$‚(:” òîŠÓ‹Âá³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚303.892' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.29733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eMþJ¬$‚(:¿  òîŠÓý–´ disk.drbd1828.in‚0 disk.drbd1510.out‚0 cpu‚25.33006629781647 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda.out‚ 391.47265625 disk.rbd1.out‚ 79.0244140625 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1226.out‚0 network.sent‚ 1.4650396e+07 disk.sda2.in‚0.03125) disk./.usagePercent‚70.51381710686854 disk.drbd2078.out‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2925.52734375$ memory_percent‚92.09710756937662 proc.cpu.threads‚17* proc.memory.percent‚11.904001235961914 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 391.47265625 disk.dm-0.in‚ 98.87109375 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1584.out‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1942.21484375 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0$ disk.nvme0n1.out‚45097.357421875! disk.dm-0.out‚45097.357421875 disk.drbd2078.in‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8586654663086 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 network.recv‚ 1.016061e+06 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1673.in‚0%–Tž­$‚(:“ ïŠÓŠÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.167' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.72233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ÀókI®$‚(:¾  ïŠÓËŸÞ³ disk.drbd1285.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51377998608423 disk.drbd1289.out‚0 disk.drbd1356.in‚0 cpu‚25.330004651154805 proc.cpu.threads‚17% disk./.usageGB‚1260.8580017089844 disk.drbd2041.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd1550.out‚0$ memory_percent‚92.09712346394858 disk.drbd1584.in‚0 disk.sda2.out‚ 392.20703125 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0! disk.dm-0.out‚45097.740234375 disk.rbd1.out‚ 79.056640625 disk.drbd1817.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1550.in‚0 network.recv‚ 1.017183e+06 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 98.87109375 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1942.2109375 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2925.52734375 network.sent‚ 1.4653424e+07 disk.drbd2078.out‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚45097.740234375 disk.drbd2118.in‚0 disk.sda.out‚ 392.20703125 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0Óá˯$‚(:’ ïŠÓé´³³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚170.24' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚56.74666666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:xk(D°$‚(:¹  ïŠÓ˜€Ü³ disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.09712346394858 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 79.056640625 disk.nvme0n1.in‚ 98.87109375 disk.drbd1289.out‚0* proc.memory.percent‚11.904001235961914 disk.drbd1938.in‚0 disk.dm-0.in‚ 98.87109375 disk.drbd2041.in‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚45097.931640625 disk.drbd1550.out‚0% disk./.usageGB‚1260.8580017089844 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 cpu‚25.32995869548334) disk./.usagePercent‚70.51377998608423 disk.drbd1817.in‚0 disk.sda2.out‚ 392.484375 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 392.484375 disk.drbd1931.out‚0 network.recv‚ 1.018376e+06! disk.dm-0.out‚45097.931640625 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 disk.drbd1226.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 1942.2109375 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 network.sent‚ 1.4656526e+07”ž3 ž±$‚(:“ ŸïŠÓÊíÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚273.545' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.18166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë‡ÃB²$‚(:·  ŸïŠÓ„Ãñ³ network.sent‚ 1.4660107e+07% disk./.usageGB‚1260.8580436706543# disk.dm-0.out‚45098.85791015625 disk.rbd1.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 network.recv‚ 1.019511e+06 disk.sda.out‚ 392.74609375 disk.drbd1673.in‚0 disk.sda2.in‚0.03125 disk.drbd1285.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.51378233280047 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚98.875& disk.nvme0n1.out‚45098.85791015625 disk.drbd1226.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 392.74609375 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 79.400390625$ memory_percent‚92.09712346394858# proc.memory.rssMB‚ 2925.52734375( proc.memory.availableMB‚ 1942.2109375 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 cpu‚25.32991218975455 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚98.875 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0¶¥¥4³$/ _wandbruntime‚15198  _runtime‚15198í6ÅXÈ´$ trainer/global_step‚8349 % _timestamp‚1.7848544351818717e+09 _runtime‚15198.539104953 _step‚693 & lr-AdamW/pg1‚4.255805850444949e-05 & lr-AdamW/pg2‚4.255805850444949e-05 ãÊ÷͵$Ç trainer/global_step‚8349 % _timestamp‚1.7848544351818717e+09 _runtime‚15198.539104953 _step‚693 & lr-AdamW/pg1‚4.255805850444949e-05 & lr-AdamW/pg2‚4.255805850444949e-05µë”(ñ4¶$/ _wandbruntime‚15198  _runtime‚15198‡^÷Ñ (·$„P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2402292612135568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22248576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8712379793920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72462386 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77512635 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47682245 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚668463820236288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚657808704796672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2411951292981248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8711034357760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚334 _step‚694 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77522699 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2411951292981248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2403240284796928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72462386 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149975021 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72462392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚15198.540420882 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77522699 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2411952664590848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77512635 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25433335 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2402292637563536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848544351829135e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72462392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚668464235400192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8712379793920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72462386 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77512635 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47682245 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚668463820236288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚657808704796672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2411951292981248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8711034357760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚334 _step‚694 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77522699 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2411951292981248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2403240284796928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72462386 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149975021 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72462392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚15198.540420882 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77522699 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2411952664590848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77512635 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25433335 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2402292637563536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 % _timestamp‚1.7848544351829135e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72462392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚668464235400192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22248576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¶¥8b-4¹$/ _wandbruntime‚15200  _runtime‚15200&•C'º$N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2402571439337312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2411268883028080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72470759 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚668543659847168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2412232596396544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2403519194087424 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2412232596396544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25436237 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚337 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2412231224779776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149992470 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2402571413909344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8698813730134 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150002554 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2403519194087424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77521717 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚668544077101056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8713402309120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77521717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77531795 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8712056865792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8713402309120 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10656736775680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72470759 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77531795 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72470753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47687448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149992470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22251211 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚332 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657886923071488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25436569 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150002554 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47687785 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10656785938432Ëb×k'»$’N REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77531795 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8712056865792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8713402309120 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10656736775680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72470759 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77531795 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72470753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47687448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149992470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22251211 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚332 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657886923071488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25436569 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150002554 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47687785 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10656785938432 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2402571439337312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2411268883028080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72470759 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚668543659847168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2412232596396544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2403519194087424 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2412232596396544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25436237 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚337 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2412231224779776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149992470 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2402571413909344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8698813730134 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150002554 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2403519194087424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77521717 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚668544077101056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8713402309120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77521717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824·ø Ey4¼$/ _wandbruntime‚15200  _runtime‚15200lq‚\½$ 0loss/FBt2/assignment_loss‚0.0489339367198882 /loss/FRt1/detection_loss‚0.5553560779479599 0loss/SRqqt2/detection_loss‚0.621958839608536 %loss/total_loss‚7.977358957337523 # _timestamp‚1.78485443669491e+09 2loss/SRqqt1/assignment_loss‚0.5155113593089578 /loss/FBt2/detection_loss‚0.4797383860837172 trainer/global_step‚8349 0loss/FRt1/assignment_loss‚1.5863102679753818 0loss/FRt2/assignment_loss‚0.6157727157762514 3loss/FBt1/assignment_loss‚0.0006996784587696508 /loss/FBt1/detection_loss‚0.6731064067775937 _step‚696 2loss/SRqqt2/assignment_loss‚0.5213631470092697 /loss/FRt2/detection_loss‚1.4275070248047366 1loss/SRqqt1/detection_loss‚0.9311011168664616 epoch‚29 _runtime‚15200.052116748ß[u« ¾$† 0loss/FRt1/assignment_loss‚1.5863102679753818 0loss/FRt2/assignment_loss‚0.6157727157762514 3loss/FBt1/assignment_loss‚0.0006996784587696508 /loss/FBt1/detection_loss‚0.6731064067775937 _step‚696 2loss/SRqqt2/assignment_loss‚0.5213631470092697 /loss/FRt2/detection_loss‚1.4275070248047366 1loss/SRqqt1/detection_loss‚0.9311011168664616 epoch‚29 _runtime‚15200.052116748 0loss/FBt2/assignment_loss‚0.0489339367198882 /loss/FRt1/detection_loss‚0.5553560779479599 0loss/SRqqt2/detection_loss‚0.621958839608536 %loss/total_loss‚7.977358957337523 # _timestamp‚1.78485443669491e+09 2loss/SRqqt1/assignment_loss‚0.5155113593089578 /loss/FBt2/detection_loss‚0.4797383860837172 trainer/global_step‚8349¸VË̞¿$‚(:“ ®ïŠÓà à³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚283.633' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.54433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿtû†AÀ$‚(:¶  ®ïŠÓòÁý³ disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 cpu‚25.329882418266365 disk.drbd1226.in‚0 disk.dm-0.in‚98.875 disk.drbd1673.in‚0 disk.drbd1289.out‚0 network.recv‚ 1.020511e+06 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0& disk.nvme0n1.out‚45099.17431640625$ memory_percent‚92.09726651509604 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.904001235961914 disk.drbd2041.out‚0# disk.dm-0.out‚45099.17431640625 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.sda1.in‚0 disk.drbd1356.out‚0% disk./.usageGB‚1260.8579711914062 disk.sda.out‚ 393.0390625 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.sda2.out‚ 393.0390625 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1817.out‚0 disk.drbd1828.out‚0( disk./.usagePercent‚70.5137782793815 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0 network.sent‚ 1.4662554e+07 disk.drbd1931.out‚0 disk.nvme0n1.in‚98.875 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1942.17578125 disk.drbd2078.in‚0 disk.rbd1.out‚ 79.400390625 disk.drbd1510.in‚0Ž-ÊÁ$‚(:’ ½ïŠÓÉøij gpu.0.gpu‚16.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚291.059' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.01966666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯„ 6›Â$‚(:Ä  ½ïŠÓÔàæ³ disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚45099.51806640625 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sda1.in‚0$ memory_percent‚92.09726651509604 disk.drbd1289.out‚0# proc.memory.rssMB‚ 2925.52734375 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1942.17578125 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 101.6484375& disk.nvme0n1.out‚45099.51806640625* proc.memory.percent‚11.904001235961914 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1550.out‚0 cpu‚25.329820898790654 disk.drbd2111.out‚0 disk.drbd1552.out‚0 network.recv‚ 1.023369e+06) disk./.usagePercent‚70.51377870605718 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 79.4326171875 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚Þ®A1´0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 393.58203125 network.sent‚ 1.4722628e+07 disk.drbd1673.in‚0 disk.drbd2041.out‚0% disk./.usageGB‚1260.8579788208008 disk.dm-0.in‚ 101.6484375 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 393.58203125 disk.drbd1817.out‚0ø÷¸êžÃ$‚(:“ ÌïŠÓѺܳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚283.367' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.45566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\µ8½MÄ$‚(:  ÌïŠÓµÄý³ disk.drbd2111.in‚0 disk.sda.out‚ 394.13671875 network.sent‚ 1.4725614e+07 disk.nvme0n1.in‚ 101.6484375& disk.nvme0n1.out‚45099.81298828125 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0$ memory_percent‚92.09739367167154 disk.drbd1550.in‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 1942.14453125 network.recv‚ 1.024446e+06 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51377977274639 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0$ disk./.usageGB‚1260.857997894287 disk.drbd1289.out‚0 disk.dm-0.in‚ 101.6484375# disk.dm-0.out‚45099.81298828125 disk.drbd1673.in‚0 disk.drbd1828.in‚0 cpu‚25.32974188950688 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.sda2.out‚ 394.13671875 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 79.4326171875 disk.drbd1931.in‚0 disk.sda.in‚0.03125 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2925.52734375xYÆ}žÅ$‚(:“ ÛïŠÓÜèÀ³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.417' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.13899999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°7ÖyLÆ$‚(:Á  ÛïŠÓÐæܳ disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8579216003418 disk.sda.in‚0.03125 disk.sda.out‚ 394.33203125 disk.drbd2118.in‚0# memory_percent‚92.0974095662435) disk./.usagePercent‚70.51377550598957 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1942.140625# proc.memory.rssMB‚ 2925.52734375 disk.drbd2041.in‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚45100.70458984375 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0* proc.memory.percent‚11.904001235961914 disk.sda2.out‚ 394.33203125 disk.rbd1.in‚0 disk.rbd1.out‚ 79.5341796875 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd1285.in‚0 cpu‚25.329713595724964 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚45100.70458984375 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 network.sent‚ 1.4728618e+07 network.recv‚ 1.025383e+06 disk.drbd1965.out‚0 disk.drbd1285.out‚0 æåìŸÇ$‚(:” êïŠÓíØå³ gpu.0.gpu‚35.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚108.379' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.126333333333335 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aeœHÈ$‚(:½  êïŠÓ£·‡´ disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1356.in‚0' proc.memory.availableMB‚ 1942.140625 proc.cpu.threads‚17 disk.sda.out‚ 394.984375 disk.drbd2111.out‚0# memory_percent‚92.0974095662435% disk./.usageGB‚1260.8579406738281 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚45101.13037109375 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 network.sent‚ 1.4731663e+07 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1673.out‚0 cpu‚25.329666979023276 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 394.984375 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd2118.out‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.904001235961914 disk.rbd1.out‚ 79.5966796875 disk.drbd1510.in‚0 disk.sda.in‚0.03125# disk.dm-0.out‚45101.13037109375 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2925.52734375 network.recv‚ 1.026508e+06) disk./.usagePercent‚70.51377657267878 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0óJgSžÉ$‚(:“ ùïŠÓÃÄÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.128' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.37599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lõ¯GÊ$‚(:¼  ùïŠÓ˜Éû³ disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 395.53515625 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0( disk./.usagePercent‚70.5138843082884 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd2041.in‚0 disk.drbd1285.out‚0 network.sent‚ 1.4734665e+07 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.dm-0.in‚ 101.6484375# proc.memory.rssMB‚ 2925.52734375 disk.drbd2111.out‚0 disk.rbd1.out‚ 79.5966796875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.sda1.out‚0 network.recv‚ 1.027446e+06 disk.sda.out‚ 395.53515625 disk.drbd1965.out‚0% disk./.usageGB‚1260.8598670959473 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.dm-0.out‚45101.98828125# disk.nvme0n1.out‚45101.98828125 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.904001235961914 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd1938.out‚0$ memory_percent‚92.09437370300293 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1942.88671875 disk.drbd1817.in‚0 disk.drbd1931.out‚0 cpu‚25.32958929061629q‡4Ë$/ _wandbruntime‚15285  _runtime‚15285ˆë×MÈÌ$ & lr-AdamW/pg1‚4.191414861199555e-05 & lr-AdamW/pg2‚4.191414861199555e-05 trainer/global_step‚8399 % _timestamp‚1.7848545220984988e+09 _runtime‚15285.455740401 _step‚697t¤„öÍÍ$Ç & lr-AdamW/pg1‚4.191414861199555e-05 & lr-AdamW/pg2‚4.191414861199555e-05 trainer/global_step‚8399 % _timestamp‚1.7848545220984988e+09 _runtime‚15285.455740401 _step‚697¹Ü2VÓ4Î$/ _wandbruntime‚15285  _runtime‚15285Hœ¨è (Ï$ƒP YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150859393 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2416232700824368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672455841888768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77968587 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2424981623063526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2417185723148288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47960993 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10718238152704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8763569350144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2417185749321728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2416232726252336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150849323 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784854522099559e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚15285.460945499 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47960641 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8747552199904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2417185749321728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72880736 trainer/global_step‚8399 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8762223913984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _step‚698 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72880742 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25580417 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672456261246976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8748896811190 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72880742 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661738023094272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8763569350144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2425947947062272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2417185723148288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22380576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661737655003136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8762223913984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _step‚698 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72880742 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25580417 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672456261246976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8748896811190 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72880742 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661738023094272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8763569350144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2425947947062272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2417185723148288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22380576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661737655003136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2416232700824368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672455841888768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77968587 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2424981623063526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2417185723148288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47960993 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10718238152704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8763569350144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2417185749321728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2416232726252336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150849323º™³ÜØ4Ñ$/ _wandbruntime‚15286  _runtime‚15286:Þpüì Ò$N XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8748573001544 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661816241369088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72889103 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8764591865344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47966210 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2416511502598144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8763246422016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 172805†&x'3248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2417464632438784 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 _runtime‚15286.933018591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672535685776896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25583348 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2417464658612224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77977669 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150876856 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2417464658612224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2426227878860800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150866772 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2425260075599688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10719444407808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661816609460224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77987747 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582999 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8763246422016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77987747 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22383216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672536107225088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚699 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150866772 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72889109 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2426229250477568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22383211 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10719497764864xµß¼'Ó$’N RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 _runtime‚15286.933018591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672535685776896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25583348 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2417464658612224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77977669 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150876856 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2417464658612224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2426227878860800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150866772 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2425260075599688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10719444407808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661816609460224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77987747 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582999 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8763246422016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77987747 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22383216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672536107225088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚699 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150866772 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72889109 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2426229250477568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22383211 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10719497764864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8748573001544 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661816241369088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72889103 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8764591865344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47966210 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2416511502598144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8763246422016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2417464632438784 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56»Ä‚ûŒ4Ô$/ _wandbruntime‚15286  _runtime‚15286Àmy1 Õ$… 2loss/FBt2/assignment_loss‚0.010910448130519867 /loss/FBt1/detection_loss‚0.6822797321670112 .loss/FBt2/detection_loss‚0.466082641581953 0loss/FRt2/assignment_loss‚0.6340664661826739 3loss/SRqqt2/assignment_loss‚0.47531483466993657 /loss/FRt1/detection_loss‚0.5530562174020569 % _timestamp‚1.7848545236122575e+09 _runtime‚15286.96947595 /loss/FRt2/detection_loss‚1.3848411568476562 1loss/SRqqt2/detection_loss‚0.6688745152350316 %loss/total_loss‚7.842208530191026 _step‚700 0loss/FRt1/assignment_loss‚1.5975284265334326 2loss/SRqqt1/assignment_loss‚0.5383290967530329 3loss/FBt1/assignment_loss‚0.0042603958059418785 1loss/SRqqt1/detection_loss‚0.8266645988817791 epoch‚29 trainer/global_step‚8399+ÎIÖ$Š 0loss/FRt2/assignment_loss‚0.6340664661826739 3loss/SRqqt2/assignment_loss‚0.47531483466993657 /loss/FRt1/detection_loss‚0.5530562174020569 % _timestamp‚1.7848545236122575e+09 _runtime‚15286.96947595 /loss/FRt2/detection_loss‚1.3848411568476562 1loss/SRqqt2/detection_loss‚0.6688745152350316 %loss/total_loss‚7.842208530191026 _step‚700 0loss/FRt1/assignment_loss‚1.5975284265334326 2loss/SRqqt1/assignment_loss‚0.5383290967530329 3loss/FBt1/assignment_loss‚0.0042603958059418785 1loss/SRqqt1/detection_loss‚0.8266645988817791 epoch‚29 trainer/global_step‚8399 2loss/FBt2/assignment_loss‚0.010910448130519867 /loss/FBt1/detection_loss‚0.6822797321670112 .loss/FBt2/detection_loss‚0.466082641581953¼©*†›>×$‚(:³  ˆðŠÓЩõ³ disk.drbd1584.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1941.84765625 cpu‚25.33154960660899% disk./.usageGB‚1260.8589401245117 disk.sda2.in‚0.03125 disk.dm-0.out‚ 45102.765625# proc.memory.rssMB‚ 2926.51171875 disk.drbd1828.out‚0! disk.nvme0n1.out‚ 45102.765625 disk.drbd1673.in‚0" memory_percent‚92.098601659139 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.sda.out‚ 395.89453125 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 network.sent‚ 1.473766e+07 network.recv‚ 1.028555e+06 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1938.out‚0) proc.memory.percent‚11.90800666809082 disk.drbd2118.in‚0 disk.sda2.out‚ 395.89453125 disk.drbd1289.in‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 79.72265625 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.51383246719308K¨’o“Ø$‚(:ˆ ˆðŠÓ‰¸Ö³ gpu.0.gpu‚30.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.746' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.582 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯µÔ4Ù$/ _wandbruntime‚15306  _runtime‚15306|9‚Ñ./Ú$èu @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 trainer/global_step‚8399 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6147180795669556 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15213048458099365 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3826139569282532 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 *jet/accuracy_1_of_2‚0.7771931290626526 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6409835815429688 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3619629442691803 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8689728379249573 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1388361155986786 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24452747404575348 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8195809721946716 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21581535041332245 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.735624372959137 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6474939584732056 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3553977310657501 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9182175993919373 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7099055051803589 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09077607095241547 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22642290592193604 +jet/accuracy_2_of_2‚0.16384169459342957 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6944106817245483 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8173831701278687 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4510354995727539 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23971572518348694 _runtime‚15306.372117536 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2999212443828583 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2124645709991455 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.255313903093338 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_1‚1 _step‚701 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26741719245910645 'particle/f_score‚0.6551251411437988 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8895537257194519 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7563702464103699 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5830821990966797 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7313184142112732 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/accuracy‚0.854242205619812 % _timestamp‚1.7848545430140388e+09 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 /particle/accuracy_4_of_4‚0.9640940427780151 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6310542821884155 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6919117569923401 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 *loss/val_total_loss‚7.8767354830760015 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.687446117401123 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04611920937895775 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7929742932319641 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 +particle/specificity‚0.9113420248031616 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7277754545211792 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6289820075035095 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8659297227859497 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8179653882980347 7validation_average_jet_accuracy‚0.39121338725090027 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 -"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9940206408500671 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06285373866558075 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23425868153572083 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3033931255340576 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 +particle/sensitivity‚0.6452158689498901 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3347841799259186 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.674734890460968 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6544328927993774 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49846503138542175 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2524511218070984 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12333177030086517 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.32276803255081177 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42587828636169434 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028435487300157547 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22630678117275238 -"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13646124303340912 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30859389901161194 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0307844877243042 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24871888756752014 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7247908711433411 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6905983686447144 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6384372115135193 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7294002771377563 *jet/accuracy_1_of_3‚0.9256716966629028 *jet/accuracy_1_of_4‚0.9743425250053406 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0379735566675663 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32818007469177246 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917822480201721 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6708094477653503 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6129966974258423 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8163419961929321 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07354454696178436 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978915452957153 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7889249920845032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7661904096603394 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30899837613105774 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26527222990989685 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45184439420700073 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7611981630325317 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.645534098148346 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25205618143081665 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8114718794822693 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_2_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.022002996876835823 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7984914183616638 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12479162216186523 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7066227793693542 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023681338876485825 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8895537257194519 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5522547960281372 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_4‚0.8529815673828125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9390110373497009 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38130345940589905 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997065007686615 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8109301924705505 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.00090680801076•¸ñÀ 81453 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4888129234313965 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5260815024375916 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7272838354110718 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7097547650337219 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6735437512397766 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04531082138419151 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8096316456794739 epoch‚29 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5919180512428284 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.10971427708864212 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.362291157245636 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6495797038078308 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30899837613105774 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 particle/accuracy_3_of_4‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6986908912658691 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32160839438438416 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7270389795303345 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_2‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32107266783714294 *jet/accuracy_2_of_3‚0.5943474769592285 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.645534098148346 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547(ùÃó:Û$íu =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30899837613105774 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26527222990989685 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45184439420700073 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7611981630325317 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.645534098148346 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25205618143081665 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8114718794822693 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_2_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.022002996876835823 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925479292869568 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7984914183616638 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12479162216186523 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7066227793693542 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023681338876485825 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8895537257194519 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5522547960281372 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_4‚0.8529815673828125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9390110373497009 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38130345940589905 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997065007686615 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8109301924705505 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4888129234313965 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5260815024375916 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7272838354110718 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7097547650337219 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6735437512397766 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04531082138419151 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8096316456794739 epoch‚29 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5919180512428284 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.10971427708864212 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.362291157245636 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6495797038078308 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30899837613105774 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 particle/accuracy_3_of_4‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6986908912658691 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32160839438438416 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9947624206542969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7270389795303345 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_2‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32107266783714294 *jet/accuracy_2_of_3‚0.5943474769592285 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.645534098148346 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 trainer/global_step‚8399 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6147180795669556 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15213048458099365 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3826139569282532 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 *jet/accuracy_1_of_2‚0.7771931290626526 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6409835815429688 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3619629442691803 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8689728379249573 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1388361155986786 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24452747404575348 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8195809721946716 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21581535041332245 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.735624372959137 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6474939584732056 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3553977310657501 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9182175993919373 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7099055051803589 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09077607095241547 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22642290592193604 +jet/accuracy_2_of_2‚0.16384169459342957 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6944106817245483 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8173831701278687 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4510354995727539 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23971572518348694 _runtime‚15306.372117536 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2999212443828583 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2124645709991455 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.255313903093338 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_1‚1 _step‚701 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26741719245910645 'particle/f_score‚0.6551251411437988 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8895537257194519 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7563702464103699 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5830821990966797 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7313184142112732 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/accuracy‚0.854242205619812 % _timestamp‚1.7848545430140388e+09 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 /particle/accuracy_4_of_4‚0.9640940427780151 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6310542821884155 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6919117569923401 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 *loss/val_total_loss‚7.8767354830760015 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.687446117401123 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04611920937895775 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7929742932319641 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 +particle/specificity‚0.9113420248031616 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7277754545211792 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6289820075035095 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8659297227859497 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8179653882980347 7validation_average_jet_accuracy‚0.39121338725090027 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 -"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9940206408500671 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06285373866558075 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23425868153572083 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3033931255340576 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 +particle/sensitivity‚0.6452158689498901 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3347841799259186 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.674734890460968 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6544328927993774 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49846503138542175 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2524511218070984 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12333177030086517 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.32276803255081177 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42587828636169434 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028435487300157547 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22630678117275238 -"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9479166865348816 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13646124303340912 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30859389901161194 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0307844877243042 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24871888756752014 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7247908711433411 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6905983686447144 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6384372115135193 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7294002771377563 *jet/accuracy_1_of_3‚0.9256716966629028 *jet/accuracy_1_of_4‚0.9743425250053406 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0379735566675663 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32818007469177246 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917822480201721 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6708094477653503 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6129966974258423 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8163419961929321 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07354454696178436 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978915452957153 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7889249920845032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7661904096603394 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844½ ð¥-HÜ$‚(:½  —ðŠÓýÅÿ³ disk.drbd1285.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd2118.out‚0 disk.sda.out‚ 396.43359375 disk.drbd1289.out‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.dm-0.in‚ 101.6484375% disk./.usageGB‚1260.8609504699707 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 396.43359375 disk.drbd1938.in‚0" disk.dm-0.out‚45103.2080078125 disk.sda.in‚0.03125$ memory_percent‚92.11483001708984 disk.drbd2078.out‚0 disk.rbd1.out‚ 79.82421875% disk.nvme0n1.out‚45103.2080078125 disk.drbd2118.in‚0* proc.memory.percent‚11.924123764038086 disk.drbd1828.in‚0 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 cpu‚25.332609802729767# proc.memory.rssMB‚ 2930.47265625( disk./.usagePercent‚70.5139448962352 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 network.recv‚ 1.031227e+06 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1965.out‚0 network.sent‚ 1.4797679e+07 disk.sda1.in‚0' proc.memory.availableMB‚ 1937.859375”……qžÝ$‚(:“ —ðŠÓúâß³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚286.781' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.59366666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Q©žÞ$‚(:“ ¦ðŠÓ¡äá³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.616' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.53866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šwÚ™?ß$‚(:´  ¦ðŠÓËÐþ³ disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0$ memory_percent‚92.11483001708984 disk.drbd1552.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2930.47265625* proc.memory.percent‚11.924123764038086 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 80.06640625 cpu‚25.33257799532636) disk./.usagePercent‚70.51394916299202 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0$ disk./.usageGB‚1260.861026763916 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 network.sent‚ 1.4851593e+07 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚396.8125 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 101.6484375% disk.nvme0n1.out‚45103.3759765625 network.recv‚ 1.033834e+06 disk.sda.out‚396.8125 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd2111.in‚0' proc.memory.availableMB‚ 1937.859375 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda.in‚0.03125 disk.sda2.in‚0.03125 proc.cpu.threads‚17" disk.dm-0.out‚45103.3759765625 disk.drbd1285.out‚0Ì»aFà$‚(:»  µðŠÓôÀê³ disk.drbd1550.out‚0 disk.drbd1584.in‚0 network.sent‚ 1.485459e+07 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0# disk.nvme0n1.out‚45103.76953125 disk.dm-0.out‚45103.76953125 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2930.47265625 cpu‚25.332530507774134 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda.out‚ 397.15234375 disk.drbd2041.out‚0 disk.sda2.out‚ 397.15234375$ memory_percent‚92.11490948994955% disk./.usageGB‚1260.8610076904297 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125) proc.memory.availableMB‚ 1937.83984375 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.924123764038086 disk.nvme0n1.in‚ 101.6484375 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 network.recv‚ 1.034702e+06 disk.dm-0.in‚ 101.6484375 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51394809630281 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.rbd1.out‚ 80.06640625 disk.drbd1285.out‚0‘\žá$‚(:“ µðŠÓ¿Îdz gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚110.933' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.97766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w8ªªžâ$‚(:“ ÄðŠÓþ«º³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚269.828' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.94266666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JZ´¤Bã$‚(:·  ÄðŠÓ”Áà³ disk.drbd2078.out‚0" disk.nvme0n1.out‚ 45103.9296875 disk.drbd1552.out‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1937.64453125 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 101.6484375 cpu‚25.33251531468492 disk.drbd2118.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 397.4609375 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51394937632986% disk./.usageGB‚1260.8610305786133 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 network.recv‚ 1.035839e+06 disk.sda2.in‚0.03125 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 397.4609375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2930.47265625 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0$ memory_percent‚92.11570421854654* proc.memory.percent‚11.924123764038086 disk.drbd1510.in‚0 network.sent‚ 1.4857621e+07 disk.rbd1.out‚ 80.12890625 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1584.out‚0 disk.dm-0.out‚ 45103.9296875x9áGžä$‚(:“ ÓðŠÓÐô¹³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚209.003' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚69.66766666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_üîIå$‚(:¾  ÓðŠÓÎܳ) disk./.usagePercent‚70.51394830964067 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.11651484171549 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚45104.33251953125 disk.sda2.out‚ 397.8203125 disk.rbd1.out‚ 80.23046875 disk.nvme0n1.in‚ 101.6484375( proc.memory.availableMB‚ 1937.4453125 network.recv‚ 1.036934e+06 disk.drbd1584.out‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2930.47265625 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.924123764038086 disk.sda2.in‚0.03125 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 397.8203125# disk.dm-0.out‚45104.33251953125 network.sent‚ 1.4860686e+07$ disk./.usageGB‚1260.861011505127 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 cpu‚25.332500018527483 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0„'S“æ$‚(:ˆ âðŠÓŒ¬Ý³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚298.296' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I6ásMç$‚(:  âðŠÓ¦¬‚´ disk.rbd1.out‚ 80.29296875& disk.nvme0n1.out‚45104.58642578125 disk.drbd1510.in‚0 disk.sda.in‚0.03125 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2930.47265625 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0# disk.dm-0.out‚45104.58642578125* proc.memory.percent‚11.924123764038086 cpu‚25.332418760920287 network.sent‚ 1.4864218e+07 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1937.25390625 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 101.6484375 disk.nvme0n1.in‚ 101.6484375 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.03125 disk.drbd1931.out‚0$ memory_percent‚92.11729367574057 disk.drbd2111.out‚0 disk.sda.out‚ 398.18359375 disk.drbd1965.out‚0 disk.drbd1938.out‚0 network.recv‚ 1.038008e+06% disk./.usageGB‚1260.8610305786133 disk.sda2.out‚ 398.18359375 proc.cpu.threads‚17) disk./.usagePercent‚70.51394937632986 disk.drbd1828.in‚0 disk.rbd1.in‚0C½¨4è$/ _wandbruntime‚15391  _runtime‚15391©¬Êé$Ä trainer/global_step‚8449 % _timestamp‚1.7848546285690897e+09 _runtime‚15391.926363957 _step‚702 ' lr-AdamW/pg1‚4.1272421200640074e-05 ' lr-AdamW/pg2‚4.1272421200640074e-05YòÏê$É ' lr-AdamW/pg1‚4.1272421200640074e-05 ' lr-AdamW/pg2‚4.1272421200640074e-05 trainer/global_step‚8449 % _timestamp‚1.7848546285690897e+09 _runtime‚15391.926363957 _step‚702¾·-4ë$/ _wandbruntime‚15391  _runtime‚15391³byì$ƒP JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2442510896304640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2441537425625611 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73388524 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8804264922021 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2432731790664336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2442512267914240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚676504808936448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8820377342976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25754244 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78471846 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2442512267914240 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2433691890571264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25753909 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚15391.930938251 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73388518 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78481910 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48273372 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10788468031488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73388518 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151860364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8820377342976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48273372 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10788468031488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73388518 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151860364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8820377342976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2442510896304640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2441537425625611 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73388524 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8804264922021 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2432731790664336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528¿»Dbs4î$/ _wandbruntime‚15393  _runtime‚15393m®/ 'ï$ŽN J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚665794559179776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8821399858176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2433010592438112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151887897 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2442792199719936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8805285723661 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151877813 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73396891 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48278978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2441817248201139 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2442792199719936 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2433970773688320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78480928 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2441815878161773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78480928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25757203 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25756875 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676584645673472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10789667142656 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22522103 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10789718402560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2442790828103168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48279311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7848546300463846e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8820054414848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _runtime‚15393.404143142 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2433970799861760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78491006 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _runtime‚15393.404143142 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2433970799861760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78491006 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151887897 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2442792199719936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8805285723661 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.`çü"freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151877813 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73396891 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48278978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2441817248201139 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2442792199719936 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2433970773688320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78480928 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2441815878161773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78480928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25757203 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25756875 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676584645673472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10789667142656 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22522103 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10789718402560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2442790828103168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48279311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7848546300463846e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8820054414848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73806868 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48552814 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚15478.702715763 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8871566899200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73806874 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78937862 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚680496087860224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8855368804829 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2456508921995264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7848547153452919e+09 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 trainer/global_step‚8499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2446671879353136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152734666 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2456507550385664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10850845918720 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8871566899200 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2447637355096064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2456507550385664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22651468 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669645291111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73806868 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10850796748800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8856713416115 _step‚707 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73806868 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48552814 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚15478.702715763 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8871566899200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73806874 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78937862 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚680496087860224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8855368804829 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2456508921995264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7848547153452919e+09 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 trainer/global_step‚8499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2446671879353136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152734666 S>DeviceStatsMo¯˜~(…nitor.on_train_batch_start/active_bytes.all.freed‚2456507550385664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10850845918720 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8871566899200 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2447637355096064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2456507550385664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22651468 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669645291111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73806868 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10850796748800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8856713416115 _step‚707 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2447916238213120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78946958 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2456787482184192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚355 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2455807070733381 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2456788853800960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73815235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8857734217867 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25904641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25904286 _runtime‚15480.178129152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152752115 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2447916238213120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78936880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48558749 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2446950706554880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2456788853800960 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48558389 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73815235 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78936880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152762199 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8871243971072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864ê×1¥'ˆ%“N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73815235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8857734217867 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25904641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25904286 _runtime‚15480.178129152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152752115 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2447916238213120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78936880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48558749 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2446950706554880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2456788853800960 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48558389 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73815235 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78936880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152762199 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8871243971072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669724245568512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680575925359616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2447916238213120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78946958 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2456787482184192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚355 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2455807070733381 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2456788853800960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Ä™’4‰%/ _wandbruntime‚15480  _runtime‚15480í]½Š%€ .loss/FRt1/detection_loss‚0.524738844293185 /loss/FRt2/detection_loss‚1.3583985337333688 _runtime‚15480.214005907 3loss/SRqqt1/assignment_loss‚0.42081741652778026 %loss/total_loss‚8.201215429682883 trainer/global_step‚8499 2loss/SRqqt2/assignment_loss‚0.4439484875887652 0loss/SRqqt1/detection_loss‚0.888877799266395 epoch‚30 _step‚709 .loss/FRt1/assignment_loss‚1.67021873243234 0loss/FRt2/assignment_loss‚0.6483482879491154 1loss/SRqqt2/detection_loss‚0.6210125303258375 /loss/FBt1/detection_loss‚0.8910126101691276 /loss/FBt2/detection_loss‚0.7053286802447161 % _timestamp‚1.7848547168567538e+09 2loss/FBt1/assignment_loss‚0.012416236451226373 0loss/FBt2/assignment_loss‚0.0160972707010258DÍh ‹%… epoch‚30 _step‚709 .loss/FRt1/assignment_loss‚1.67021873243234 0loss/FRt2/assignment_loss‚0.6483482879491154 1loss/SRqqt2/detection_loss‚0.6210125303258375 /loss/FBt1/detection_loss‚0.8910126101691276 /loss/FBt2/detection_loss‚0.7053286802447161 % _timestamp‚1.7848547168567538e+09 2loss/FBt1/assignment_loss‚0.012416236451226373 0loss/FBt2/assignment_loss‚0.0160972707010258 .loss/FRt1/detection_loss‚0.524738844293185 /loss/FRt2/detection_loss‚1.3583985337333688 _runtime‚15480.214005907 3loss/SRqqt1/assignment_loss‚0.42081741652778026 %loss/total_loss‚8.201215429682883 trainer/global_step‚8499 2loss/SRqqt2/assignment_loss‚0.4439484875887652 0loss/SRqqt1/detection_loss‚0.888877799266395Å‹ïÓ‡GŒ%‚(:¼  ËñŠÓš¤å³ disk.drbd2041.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1552.in‚0 disk.drbd1510.in‚0 disk.sda2.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51394617626225 disk.dm-0.in‚ 101.6484375 disk.drbd1965.in‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚45107.021484375 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 80.802734375 disk.sda1.in‚0 disk.drbd2118.in‚0! disk.dm-0.out‚45107.021484375 disk.drbd1828.out‚0 disk.sda1.out‚0% disk./.usageGB‚1260.8609733581543) proc.memory.availableMB‚ 1936.26171875 disk.sda.out‚ 400.8203125 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2930.47265625 cpu‚25.33202321429242 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 network.recv‚ 1.048337e+06 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.924123764038086 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 400.8203125 disk.drbd1965.out‚0 disk.drbd1510.out‚0 network.sent‚ 1.4988194e+07 disk.sda.in‚0.03125 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0$ memory_percent‚92.12133089701334 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17îÿL!ž%‚(:“ ËñŠÓÜŽ’´ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚265.244' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.41466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kkd›Ž%‚(: ÚñŠÓì궳 gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.94' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.31333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]zèo%‚(:¿  ÚñŠÓçÜÖ³ disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1936.04296875) disk./.usagePercent‚70.51394724295146 disk.sda.in‚0.03125 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0# proc.memory.rssMB‚ 2930.47265625 network.recv‚ 1.050823e+06% disk./.usageGB‚1260.8609924316406 disk.sda2.in‚0.03125 diœø/sk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.924123764038086 network.sent‚ 1.5037784e+07 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 80.865234375 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0$ memory_percent‚92.12222099304199 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 401.12890625 disk.drbd1938.out‚0! disk.dm-0.out‚45107.216796875 cpu‚25.331928363066222 disk.sda.out‚ 401.12890625 disk.drbd1289.in‚0$ disk.nvme0n1.out‚45107.216796875 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 101.6484375 proc.cpu.threads‚17R†±¦ž%‚(:“ éñŠÓñÍé³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚269.333' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.77766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ”ÂN‘%‚(:à  éñŠÓ†üŒ´ disk.sda1.out‚0% disk./.usageGB‚1260.8597145080566 disk.drbd1552.out‚0 disk.sda1.in‚0# disk.dm-0.out‚45107.56494140625 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 80.865234375 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 401.66015625 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.12241172790527 disk.drbd2111.in‚0 disk.sda2.in‚0.03125 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 101.6484375 network.sent‚ 1.5041316e+07 disk.drbd1550.out‚0 disk.drbd2111.out‚0) proc.memory.availableMB‚ 1935.99609375) disk./.usagePercent‚70.51387577477477 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 cpu‚25.331880094739507 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2930.47265625 disk.drbd1931.out‚0 network.recv‚ 1.051847e+06 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 401.66015625 disk.drbd1289.out‚0 disk.dm-0.in‚ 101.6484375* proc.memory.percent‚11.924123764038086 disk.drbd1584.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚45107.56494140625–‡Tx‘’%‚(:† øñŠÓ¤ÁÒ³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚232.23' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚77.41 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0môlÉK“%‚(:À  øñŠÓÊÛð³ disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0" disk.dm-0.out‚45107.8701171875 disk.drbd2111.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1935.9921875 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51387470808557% disk./.usageGB‚1260.8596954345703 disk.sda.in‚0.03125 disk.drbd2078.out‚0 disk.rbd1.out‚ 80.8974609375 disk.drbd1584.in‚0 network.sent‚ 1.5043816e+07 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 401.98046875 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 network.recv‚ 1.052996e+06 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0# memory_percent‚92.1224276224772 disk.drbd2118.in‚0* proc.memory.percent‚11.924123764038086 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.dm-0.in‚ 101.6484375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 101.6484375 cpu‚25.331834062696977 disk.sda.out‚ 401.98046875# proc.memory.rssMB‚ 2930.47265625 disk.sda2.in‚0.03125 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0% disk.nvme0n1.out‚45107.8701171875 disk.drbd1226.in‚0 disk.drbd2041.out‚0ƒ£áE”%‚(:º  ‡òŠÓ„ã³# memory_percent‚92.1224276224772) disk./.usagePercent‚70.51409401938595 disk.drbd1817.in‚0* proc.memory.percent‚11.924123764038086 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 network.recv‚ 1.053955e+06 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚402.4375( proc.memory.availableMB‚ 1935.9921875 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2930.47265625 disk.sda1.out‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8636169433594 disk.rbd1.out‚ 80.9990234375 disk.drbd1817.out‚0 disk.sda2.in‚0.03125 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0# disk.dm-0.out‚45108.56103515625 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.sda.in‚0.03125 cpu‚25.331787765648766 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚45108.56103515625 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 network.sent‚ 1.5046815e+07 disk.drbd1584.out‚0 disk.sda.out‚402.4375 disk.dm-0.in‚ 101.6484375 disk.drbd1965.out‚0 disk.drbd2041.in‚0ºõ Ÿ•%‚(:” ‡òŠÓ¡¦Æ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.683' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.56099999999999 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂqÆÔ4–%/ _wandbruntime‚15565  _runtime‚15565ˆH¯Ê—%Ä ' lr-AdamW/pg2‚3.9995849987169305e-05 trainer/global_step‚8549 % _timestamp‚1.7848548021683502e+09 _runtime‚15565.525613582 _step‚710 ' lr-AdamW/pg1‚3.9995849987169305e-05#Í+Ϙ%É ' lr-AdamW/pg2‚3.9995849987169305e-05 trainer/global_step‚8549 % _timestamp‚1.7848548021683502e+09 _runtime‚15565.525613582 _step‚710 ' lr-AdamW/pg1‚3.9995849987169305e-05Æ%"¯(4™%/ _wandbruntime‚15565  _runtime‚15565ÀÁêý (š%„P A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚673574977500160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2461582819620864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22783463 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2469518440729573 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8921411019264 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚673574609409024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2470504204466688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153619038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2460611993469904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74225218 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79393814 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10913261516800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74225218 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153608968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2461582819620864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79383750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26049276 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2460611968041936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8922756455424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48832744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚8549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8906472687637 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79393814 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74225224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2469519810768827 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22783463 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2469518440729573 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8921411019264 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚673574609409024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2470504204466688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153619038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2460611993469904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74225218 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79393814 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10913261516800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74225218 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153608968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2461582819620864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79383750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26049276 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2460611968041936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8922756455424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48832744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚8549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8906472687637 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79393814 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74225224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2469519810768827 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2460890769815712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚15567.004758538 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48838295 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10914510841856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2469799633344355 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚684568074707968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2470784136265216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79392832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26052187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79402910 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8922433527296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74233585 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848548036471405e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚673653563866112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocatC*Öv¦ed‚ 8908838100675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153636501 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74233591 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22786103 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153626417 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚673653195774976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10914453290496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2460890795243680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2461861728911360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 trainer/global_step‚8549 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2470785507881984 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2461861702737920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48837925 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26051822 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8922433527296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153636501 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8923778970624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5Ÿ¬8~'ž%“N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79402910 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8922433527296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74233585 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848548036471405e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚673653563866112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8908838100675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153636501 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74233591 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22786103 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153626417 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚673653195774976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10914453290496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2460890795243680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2461861728911360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 trainer/global_step‚8549 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2470785507881984 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2461861702737920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48837925 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26051822 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8922433527296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153636501 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8923778970624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2469798263304989 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2460890769815712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚15567.004758538 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48838295 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10914510841856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2469799633344355 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚684568074707968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2470784136265216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79392832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26052187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ÈÝapó4Ÿ%/ _wandbruntime‚15567  _runtime‚15567NÂþ  %„ /loss/FBt1/detection_loss‚0.3914002112512078 %loss/total_loss‚7.730791593157884 0loss/FRt1/assignment_loss‚1.7107002352865843 0loss/FRt2/assignment_loss‚0.6541677990985537 1loss/FBt2/assignment_loss‚0.02235120687245501 /loss/FRt1/detection_loss‚0.5157137392027905 1loss/SRqqt1/detection_loss‚0.8208212493188054 trainer/global_step‚8549 % _timestamp‚1.7848548036822646e+09 _runtime‚15567.039442204 2loss/SRqqt1/assignment_loss‚0.5926247509196799 2loss/SRqqt2/assignment_loss‚0.4749269155318999 2loss/FBt1/assignment_loss‚0.047586089195617784 .loss/FRt2/detection_loss‚1.486939093132446 1loss/SRqqt2/detection_loss‚0.6054489691911333 epoch‚30 _step‚713 0loss/FBt2/detection_loss‚0.40811133415671064H ¡%‰ /loss/FBt1/detection_loss‚0.3914002112512078 %loss/total_loss‚7.730791593157884 0loss/FRt1/assignment_loss‚1.7107002352865843 0loss/FRt2/assignment_loss‚0.6541677990985537 1loss/FBt2/assignment_loss‚0.02235120687245501 /loss/FRt1/detection_loss‚0.5157137392027905 1loss/SRqqt1/detection_loss‚0.8208212493188054 trainer/global_step‚8549 % _timestamp‚1.7848548036822646e+09 _runtime‚15567.039442204 2loss/SRqqt1/assignment_loss‚0.5926247509196799 2loss/SRqqt2/assignment_loss‚0.4749269155318999 2loss/FBt1/assignment_loss‚0.047586089195617784 .loss/FRt2/detection_loss‚1.486939093132446 1loss/SRqqt2/detection_loss‚0.6054489691911333 epoch‚30 _step‚713 0loss/FBt2/detection_loss‚0.40811133415671064ÉÖZ=H¢%‚(:½  –òŠÓÒ½û³ disk.drbd1938.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1356.in‚0 network.recv‚ 1.054865e+06 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.rbd1.out‚ 81.0615234375 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚45108.9814453125 disk.drbd1550.in‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2930.47265625 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.5049799e+07 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 402.8828125 disk.drbd2078.in‚0 disk.drbd2111.in‚0( disk./.usagePercent‚70.5140931660346( proc.memory.availableMB‚ 1935.9453125 disk.drbd1817.out‚0 disk.sda2.out‚ 402.8828125* proc.memory.percent‚11.924123764038086% disk./.usageGB‚1260.8636016845703 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0$ memory_percent‚92.12261835734049 cpu‚25.33170848125485 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.sda2.in‚0.03125 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 101.6484375" disk.dm-0.out‚45108.9814453125DlÁ‡Ÿ£%‚(:” –òŠÓÊøݳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚311.575' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.85833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÊ%‚(:” ¥òŠÓšéé³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.666' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.88866666666667 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÞmÄG¥%‚(:¼  ¥òŠÓËê…´) disk./.usagePercent‚70.51409209934539$ disk./.usageGB‚1260.863582611084 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0( proc.memory.availableMB‚ 1936.1953125 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 81.0615234375 disk.sda.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 403.296875" disk.dm-0.out‚45109.7060546875% disk.nvme0n1.out‚45109.7060546875 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.924123764038086# proc.memory.rssMB‚ 2930.47265625 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0$ memory_percent‚92.12160110473633 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 403.296875 disk.dm-0.in‚ 101.6484375 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 cpu‚25.331677761423528 network.sent‚ 1.5109879e+07 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 network.recv‚ 1.057764e+06 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0¨½ž¦%‚(:“ ´òŠÓŽÝÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.465' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.82166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÆpó@§%‚(:µ  ´òŠÓ®´ñ³ disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125 disk.rbd1.out‚81.1875 disk.drbd1226.in‚0 disk.drbd1285.in‚0$ memory_percent‚92.12265014648438 disk.drbd1356.in‚0$ disk./.usageGB‚1260.863582611084 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2930.47265625 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.sda.out‚ 403.6171875 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0% proc.memory.availableMB‚ 1935.9375 disk.drbd1817.in‚0 disk.drbd1938.in‚0% disk.nvme0n1.out‚45109.8505859375 disk.drbd1226.out‚0 network.recv‚ 1.058713e+06 disk.dm-0.in‚ 101.6484375" disk.dm-0.out‚45109.8505859375 disk.drbd2078.in‚0 disk.drbd1289.in‚0 network.sent‚ 1.5112873e+07 disk.drbd1817.out‚0 disk.rbd1.in‚0* proc.memory.percent‚11.924123764038086 cpu‚25.331599936999964) disk./.usagePercent‚70.51409209934539 disk.drbd1584.out‚0 disk.sda2.out‚ 403.6171875Öx5W“¨%‚(:ˆ ÃòŠÓ†Ðܳ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚269.538' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_Eî{I©%‚(:¾  ÃòŠÓ»ú³ disk.rbd1.out‚ 81.2890625& disk.nvme0n1.out‚45110.23388671875 network.recv‚ 1.059878e+06 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.03125$ memory_percent‚92.12166468302408 disk.sda.in‚0.03125 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1936.1796875 network.sent‚ 1.5116106e+07 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 cpu‚25.33155343594491 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.6484375 disk.rbd1.in‚0) disk./.usagePercent‚70.51409209934539 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 101.6484375 proc.cpu.threads‚17 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2930.47265625$ disk./.usageGB‚1260.863582611084 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.sda2.out‚ 403.94140625# disk.dm-0.out‚45110.23388671875 disk.drbd2078.in‚0* proc.memory.percent‚11.924123764038086 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 403.94140625 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0žoîsœª%‚(:‘ ÒòŠÓŠÿµ³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚270.388' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.12933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}óBŽC«%‚(:¸  ÒòŠÓ²†Ö³ disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0$ memory_percent‚92.12166468302408 disk.rbd1.out‚ 81.3515625 disk.sda.out‚404.3125 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 cpu‚25.331492250785356 disk.drbd1584.in‚0 network.recv‚ 1.060855e+06 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0# proc.memory.rssMB‚ 2930.47265625 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.03125* proc.memory.percent‚11.924123764038086 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0( proc.memory.availableMB‚ 1936.1796875 network.sent‚ 1.5119092e+07& disk.nvme0n1.out‚45110.41357421875 disk.drbd1550.out‚0 disk.sda2.out‚404.3125# disk.dm-0.out‚45110.41357421875 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8637962341309 disk.nvme0n1.in‚ 101.6484375 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51410404626448 disk.drbd1285.in‚0 disk.sda.in‚0.03125ˆ¬Éº“¬%‚(:ˆ áòŠÓôþâ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.365' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.455 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉoÖÕK­%‚(:À  áòŠÓ«°ƒ´ disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.924123764038086 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 cpu‚25.33141309935035 network.recv‚ 1.061879e+06 disk.drbd2078.in‚0& disk.nvme0n1.out‚45110.71533203125 disk.sda1.out‚0 disk.rbd1.out‚ 81.3515625 disk.drbd1285.out‚0% disk./.usageGB‚1260.8637580871582 disk.rbd1.in‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2930.47265625 disk.drbd1673.out‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1937.48046875 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚45110.71533203125 disk.dm-0.in‚ 101.6484375 disk.drbd2078.out‚0 disk.sda.out‚ 404.73046875 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.sda2.in‚0.03125 disk.drbd1226.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51410191288608 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 404.73046875$ memory_percent‚92.11637179056802 network.sent‚ 1.5122658e+07 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.sda.in‚0.03125šíœ4®%/ _wandbruntime‚15652  _runtime‚15652Š&%zȯ% & lr-AdamW/pg1‚3.936117351429239e-05 & lr-AdamW/pg2‚3.936117351429239e-05 trainer/global_step‚8599 % _timestamp‚1.7848548889902341e+09 _runtime‚15652.347467995 _step‚714Í‹2ÂÍ°%Ç & lr-AdamW/pg1‚3.936117351429239e-05 & lr-AdamW/pg2‚3.936117351429239e-05 trainer/global_step‚8599 % _timestamp‚1.7848548889902341e+09 _runtime‚15652.347467995 _step‚714ÊÈpu4±%/ _wandbruntime‚15652  _runtime‚15652=p ²%ƒP RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26196246 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚15652.351937772 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8958921181731 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22915468 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2474552082158704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8957576570445 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2484500858547712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8973946011648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2484502230157312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79849766 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚677504295797760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8972600575488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26195876 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10975456201216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 $ _timestamp‚1.784854888991299e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154483270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2483509633301181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74643568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79839702 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49111339 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74643574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688479330543616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79849766 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2475528257972224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2474552056730736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49111714 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10975402836992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2475528257972224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544eŒc(³%ˆP TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8973946011648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2484502230157312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79849766 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚677504295797760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8972600575488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26195876 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10975456201216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 $ _timestamp‚1.784854888991299e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154483270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2483509633301181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74643568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79839702 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49111339 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74643574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688479330543616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79849766 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2475528257972224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2474552056730736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49111714 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10975402836992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2475528257972224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26196246 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚15652.351937772 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8958921181731 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22915468 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2474552082158704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8957576570445 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2484500858547712ËGJŠÉ4´%/ _wandbruntime‚15653  _runtime‚15653Fç!³'µ%ŒN XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74651935 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2484780790346240 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688559583054336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49116910 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26198807 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚677582514072576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8959941983483 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2483789455876597 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2474830858504512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2484782161963008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154510803 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2475807193436160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79858862 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49117281 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8973623083520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26199173 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688559161606144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74651941 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2484782161963008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74651941 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2474830883932480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2475807193436160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8958597372085 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚15653.818709389 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8974968526848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22918108 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74651935 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22918103 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8974968526848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 trainer/global_step‚8599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2475807167262720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10976647533568 # _timestamp‚1.78485489046114e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299sw'Ò™¶%‘N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49116910 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26198807 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚677582514072576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8959941983483 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2483789455876597 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2474830858504512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2484782161963008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154510803 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2475807193436160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79858862 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49117281 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8973623083520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26199173 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688559161606144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74651941 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2484782161963008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74651941 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2474830883932480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2475807193436160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8958597372085 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚15653.818709389 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8974968526848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22918108 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74651935 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22918103 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8974968526848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 trainer/global_step‚8599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2475807167262720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10976647533568 # _timestamp‚1.78485489046114e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74651935 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2484780790346240 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688559583054336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248̳X¸L4·%/ _wandbruntime‚15653  _runtime‚15653û”é= ¸%ƒ 2loss/SRqqt2/assignment_loss‚0.5006436643976129 1loss/SRqqt1/detection_loss‚0.8671512042094643 trainer/global_step‚8599 % _timestamp‚1.7848548905021281e+09 _runtime‚15653.859307344 2loss/FBt2/assignment_loss‚0.022099852892913784 -loss/FRt2/detection_loss‚1.46905880420738 /loss/FBt1/detection_loss‚0.5017924306394402 %loss/total_loss‚7.817191377263064 1loss/FBt1/assignment_loss‚0.01498793776556134 1loss/SRqqt2/detection_loss‚0.5924402378449034 _step‚717 0loss/FRt2/assignment_loss‚0.6441635701328657 /loss/FRt1/detection_loss‚0.5283325484123147 /loss/FBt2/detection_loss‚0.4717520389761836 epoch‚30 0loss/FRt1/assignment_loss‚1.7096529466254207 3loss/SRqqt1/assignment_loss‚0.49511614115900254c­Îó¹%ˆ 1loss/FBt1/assignment_loss‚0.01498793776556134 1loss/SRqqt2/detection_loss‚0.5924402378449034 _step‚717 0loss/FRt2/assignment_loss‚0.6441635701328657 /loss/FRt1/detection_loss‚0.5283325484123147 /loss/FBt2/detection_loss‚0.4717520389761836 epoch‚30 0loss/FRt1/assignment_loss‚1.7096529466254207 3loss/SRqqt1/assignment_loss‚0.49511614115900254 2loss/SRqqt2/assignment_loss‚0.5006436643976129 1loss/SRqqt1/detection_loss‚0.8671512042094643 trainer/global_step‚8599 % _timestamp‚1.7848548905021281e+09 _runtime‚15653.859307344 2loss/FBt2/assignment_loss‚0.022099852892913784 -loss/FRt2/detection_loss‚1.46905880420738 /loss/FBt1/detection_loss‚0.5017924306394402 %loss/total_loss‚7.817191377263064Í÷û˜Ó“º%‚(:ˆ ðòŠÓۘͳ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚251.019' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.673 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Ë÷C»%‚(:¸  ðòŠÓÏ£ð³ disk.sda.out‚ 405.1953125 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 101.6484375 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.03125 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1828.in‚0 disk.drbd1285.in‚0% disk./.usageGB‚1260.8656921386719 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚81.59375) proc.memory.availableMB‚ 1937.39453125# proc.memory.rssMB‚ 2930.47265625 cpu‚25.33135171117584$ disk.nvme0n1.out‚45111.337890625 disk.drbd1226.out‚0) disk./.usagePercent‚70.51421007517138 disk.sda.in‚0.03125 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.11672147115073! disk.dm-0.out‚45111.337890625 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.924123764038086 network.sent‚ 1.5125206e+07 disk.sda2.out‚ 405.1953125 disk.drbd2111.out‚0 disk.drbd1550.out‚0 network.recv‚ 1.062963e+06 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1938.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0£$—T@¼%‚(:µ  ÿòŠÓúçç³ disk.drbd1965.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2930.47265625 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.sda2.in‚0.03125 disk.dm-0.out‚ 45112.140625 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 405.66796875 disk.drbd2041.in‚0 network.sent‚ 1.5185275e+07) disk./.usagePercent‚70.51421007517138% disk./.usageGB‚1260.8656921386719 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 cpu‚25.331306078839457 disk.sda.out‚ 405.66796875 disk.drbd2078.in‚0 network.recv‚ 1.065793e+06! disk.nvme0n1.out‚ 45112.140625$ memory_percent‚92.11672147115073 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.924123764038086 disk.drbd2118.in‚0 disk.sda.in‚0.03125 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1937.39453125 disk.rbd1.out‚81.78125 disk.drbd1550.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1552.in‚0 disk.drbd1584.out‚0§Éœqž½%‚(:“ ÿòŠÓ٥ij gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚295.958' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.65266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0* €õ9¾%‚(:®  ŽóŠÓñéî³ disk.rbd1.out‚81.84375 disk.nvme0n1.out‚ 45112.3125 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1938.in‚0' proc.memory.availableMB‚ 1937.640625 disk.rbd1.in‚0 cpu‚25.331228329389297 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 406.01953125 disk.dm-0.in‚ 101.6484375 disk.drbd1510.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2930.47265625 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.sda2.out‚ 406.01953125 disk.dm-0.out‚ 45112.3125 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.924123764038086 proc.cpu.threads‚17 network.recv‚ 1.066704e+06 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1510.in‚0 network.sent‚ 1.5188253e+07) disk./.usagePercent‚70.51421007517138% disk./.usageGB‚1260.8656921386719 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0# memory_percent‚92.1157201131185 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.03125 disk.drbd1289.out‚0\ ¯³¿%‚(:’ ŽóŠÓŽÃϳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.04' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.01333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p«ÈGžÀ%‚(:“ óŠÓΛ½³ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚254.104' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.70133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02—»@Á%‚(:µ  óŠÓæóس' proc.memory.availableMB‚ 1937.390625 disk.drbd1356.out‚0 disk.drbd1550.out‚0 network.recv‚ 1.067867e+06 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125% disk./.usageGB‚1260.8656921386719$ disk.nvme0n1.out‚45112.794921875 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 cpu‚25.331183258344357 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.03125 disk.drbd1931.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51421007517138 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.sda.out‚ 406.484375$ memory_percent‚92.11673736572266! disk.dm-0.out‚45112.794921875 disk.drbd2118.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2930.47265625 network.sent‚ 1.5191281e+07 disk.drbd1938.in‚0 disk.drbd2118.in‚0 disk.drbd1226.out‚0 disk.drbd1828.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚81.84375 disk.drbd1550.in‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.924123764038086 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.sda2.out‚ 406.484375 /ìø“Â%‚(:ˆ ¬óŠÓ—“ã³ gpu.0.gpu‚41.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚286.422' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.474 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž­XšGÃ%‚(:¼  ¬óŠÓ™Ñ€´) disk./.usagePercent‚70.51420922182001 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 network.sent‚ 1.5194262e+07 disk.drbd1552.in‚0 disk.sda.in‚0.03125 disk.sda2.out‚ 406.6796875 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1510.in‚0* proc.memory.percent‚11.924123764038086 cpu‚25.33113665847885 disk.dm-0.in‚ 101.6484375 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0! disk.dm-0.out‚45112.947265625 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1552.out‚0$ disk.nvme0n1.out‚45112.947265625( proc.memory.availableMB‚ 1937.2265625# proc.memory.rssMB‚ 2930.47265625 disk.drbd1817.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 406.6796875 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd2111.in‚0 disk.rbd1.out‚ 81.8759765625 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0$ memory_percent‚92.11740493774414 proc.cpu.threads‚17 network.recv‚ 1.068971e+06 disk.drbd2041.out‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8656768798828 disk.drbd2118.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0n¸”“Ä%‚(:ˆ »óŠÓþ¡¶³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.574' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eeC£FÅ%‚(:»  »óŠÓú£Ö³ disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0! disk.dm-0.out‚45113.302734375 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 407.078125 disk.dm-0.in‚ 101.6484375 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1356.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 81.9775390625) disk./.usagePercent‚70.51420922182001 disk.drbd2111.out‚0 disk.drbd1965.out‚0$ memory_percent‚92.11740493774414 network.recv‚ 1.070049e+06 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.924123764038086 disk.rbd1.in‚0 disk.sda.out‚ 407.078125% disk./.usageGB‚1260.8656768798828 disk.sda.in‚0.03125 disk.drbd1938.in‚0 disk.drbd2078.out‚0 network.sent‚ 1.5197291e+07 disk.nvme0n1.in‚ 101.6484375 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0$ disk.nvme0n1.out‚45113.302734375 disk.drbd1817.out‚0 disk.drbd1673.in‚0( proc.memory.availableMB‚ 1937.2265625# proc.memory.rssMB‚ 2930.47265625 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.03125 cpu‚25.331044729599807 disk.drbd1552.in‚0 disk.drbd2041.in‚09n˜4Æ%/ _wandbruntime‚15738  _runtime‚15738œž’yÆÇ%À _runtime‚15738.880077082 _step‚718 % lr-AdamW/pg1‚3.87290141590577e-05 % lr-AdamW/pg2‚3.87290141590577e-05 trainer/global_step‚8649 % _timestamp‚1.7848549755228276e+09žÈ’ÊËÈ%Å _step‚718 % lr-AdamW/pg1‚3.87290141590577e-05 % lr-AdamW/pg2‚3.87290141590577e-05 trainer/global_step‚8649 % _timestamp‚1.7848549755228276e+09 _runtime‚15738.880077082ÎÞÒd4É%/ _wandbruntime‚15738  _runtime‚15738|i (Ê%„P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2497500825872789 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155357572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2488492170847504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80295654 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚375 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚681433614095360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11037836473344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2498498884238336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2489473748670464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8649 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2498497512628736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9023790131712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23047468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2489473722497024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80305718 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49392395 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26344932 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚719 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75061918 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75061924 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80295654 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155367642 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11037785206272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9025135567872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26345307 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9008680453253 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2498497512628736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75061918 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2488492145419536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49392775 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚692471031210496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23047468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2489473722497024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80305718 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49392395 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26344932 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚719 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75061918 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75061924 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80295654 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155367642 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11037785206272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9025135567872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26345307 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9008680453253 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDevi·ÙUÇceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2498497512628736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75061918 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2488492145419536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49392775 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚692471031210496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚375 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚681433614095360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11037836473344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2498498884238336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2489473748670464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8649 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2498497512628736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9023790131712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23050103 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80304736 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155385105 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155375021 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26348326 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9011045866291 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2498777444427264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155385105 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 trainer/global_step‚8649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9024812639744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75070285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11039040520704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75070285 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9026158083072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2488770947193312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚380 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80314814 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚692551292241920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15740.358176784 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2489752657960960 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2489752631787520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26347951 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2497780648448205 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155375021 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 % _timestamp‚1.7848549770005734e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75070291 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11039091780608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49398054 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚375 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181éq„1'Î%“N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚692550872890880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15740.358176784 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2489752657960960 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2489752631787520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26347951 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2497780648448205 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155375021 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 % _timestamp‚1.7848549770005734e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75070291 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11039091780608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49398054 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚375 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2498778816044032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49398434 _step‚720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23050103 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80304736 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155385105 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155375021 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26348326 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9011045866291 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2498777444427264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155385105 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 trainer/global_step‚8649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9024812639744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75070285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11039040520704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75070285 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9026158083072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2488770947193312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚380 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80314814 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚692551292241920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32190635800361633 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6708695292472839 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6275082230567932 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_3‚0.6109134554862976 *jet/accuracy_2_of_4‚0.8307026624679565 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6279267072677612 particle/accuracy_2_of_4‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.54305100440979 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3984135687351227 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6654492020606995 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44809815287590027 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11429283767938614 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14116232097148895 7validation_average_jet_accuracy‚0.40488237142562866 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713450193405151 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6958876848220825 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5936492085456848 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12840883433818817 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762184381484985 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7590383291244507 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7765344381332397 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7213719487190247 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762184381484985 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14702899754047394 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26854220032691956 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26575422286987305 )loss/val_total_loss‚7.893551860989336 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 *jet/accuracy_1_of_4‚0.9743425250053406 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9221220016479492 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6381868124008179 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6930125951766968 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22119168937206268 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6723760366439819 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6708695292472839 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9685583114624023 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5186603665351868 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2766144573688507 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7118822932243347 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7434138059616089 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1494419276714325 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031086105853319168 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7243077158927917 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07400748878717422 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.938520073890686 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2826073467731476 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.820856511592865 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6778985857963562 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2843327820301056 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.706807017326355 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 +jet/accuracy_3_of_3‚0.07419576495885849 epoch‚30 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2696217894554138 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.342806339263916 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2724054753780365 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38032957911491394 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16211740672588348 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028721420094370842 +jet/accuracy_3_of_4‚0.10971427708864212 (particle/accuracy‚0.8540911078453064 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4482289254665375 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26204365491867065 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33090105652809143 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.603360652923584 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33090105652809143 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8085234761238098 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 'particle/f_score‚0.6559962630271912 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.806429386138916 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05702025443315506 *jet/accuracy_1_of_2‚0.7859328389167786 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8717201352119446 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7077298760414124 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +particle/specificity‚0.9102765321731567 trainer/global_step‚8679 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7658907771110535 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7413687109947205 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3434157371520996 *jet/accuracy_2_of_2‚0.1703738272190094 /particle/accuracy_3_of_3‚0.9993678331375122 +particle/sensitivity‚0.6484198570251465 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3497047424316406 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_3_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8233224749565125 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 +jet/accuracy_1_of_1‚0.37147578597068787 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2685115933418274 % _timestamp‚1.7848550485908818e+09 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6558782458305359 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚15811.94913985 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24662087857723236 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05355019494891167 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3180118501186371 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7364169955253601 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8233224749565125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3300052583217621 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8095389008522034 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02364829182624817 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.873199462890625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44646507501602173 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8196507692337036 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3440479338169098 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9418854713439941 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8017677068710327 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38249778747558594 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02196328341960907 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8226912617683411 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7211546897888184 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8122119903564453 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6383718848228455 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39985889196395874 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.508587658405304 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.734926700592041 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7321686744689941 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22472324967384338 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.044988639652729034 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 _step‚722 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6635378003120422 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7526798248291016 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5784822106361389 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6898046731948853 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6579704284667969 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7027624249458313 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7074685096740723 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34737569093704224 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675«ÎI&¿,Þ%Éu 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5186603665351868 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2766144573688507 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7118822932243347 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7434138059616089 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1494419276714325 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031086105853319168 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7243077158927917 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07400748878717422 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.938520073890686 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2826073467731476 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.820856511592865 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6778985857963562 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2843327820301056 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.706807017326355 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 +jet/accuracy_3_of_3‚0.07419576495885849 epoch‚30 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2696217894554138 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.342806339263916 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2724054753780365 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38032957911491394 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16211740672588348 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028721420094370842 +jet/accuracy_3_of_4‚0.10971427708864212 (particle/accuracy‚0.8540911078453064 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4482289254665375 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.26204365491867065 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.33090105652809143 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.603360652923584 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.33090105652809143 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8085234761238098 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 'particle/f_score‚0.6559962630271912 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.806429386138916 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05702025443315506 *jet/accuracy_1_of_2‚0.7859328389167786 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8717201352119446 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7077298760414124 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +particle/specificity‚0.9102765321731567 trainer/global_step‚8679 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7658907771110535 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7413687109947205 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3434157371520996 *jet/accuracy_2_of_2‚0.1703738272190094 /particle/accuracy_3_of_3‚0.9993678331375122 +particle/sensitivity‚0.6484198570251465 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3497047424316406 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_3_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8233224749565125 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 +jet/accuracy_1_of_1‚0.37147578597068787 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2685115933418274 % _timestamp‚1.7848550485908818e+09 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6558782458305359 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚15811.94913985 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24662087857723236 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05355019494891167 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3180118501186371 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7364169955253601 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8233224749565125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3300052583217621 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8095389008522034 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02364829182624817 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.873199462890625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44646507501602173 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8196507692337036 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3440479338169098 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9418854713439941 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8017677068710327 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38249778747558594 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02196328341960907 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8226912617683411 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7211546897888184 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8122119903564453 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6383718848228455 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39985889196395874 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.508587658405304 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.734926700592041 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7321686744689941 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22472324967384338 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.044988639652729034 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 _step‚722 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6635378003120422 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7526798248291016 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5784822106361389 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6898046731948853 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6579704284667969 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7027624249458313 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7074685096740723 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34737569093704224 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.251803994178772 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 particle/accuracy_2_of_3‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32190635800361633 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6708695292472839 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6275082230567932 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_3‚0.6109134554862976 *jet/accuracy_2_of_4‚0.8307026624679565 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6279267072677612 particle/accuracy_2_of_4‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.54305100440979 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ‡ô9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3984135687351227 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6654492020606995 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44809815287590027 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7727272510528564 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11429283767938614 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14116232097148895 7validation_average_jet_accuracy‚0.40488237142562866 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713450193405151 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6958876848220825 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5936492085456848 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12840883433818817 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762184381484985 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7590383291244507 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7765344381332397 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7213719487190247 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762184381484985 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7727272510528564 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14702899754047394 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26854220032691956 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26575422286987305 )loss/val_total_loss‚7.893551860989336 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 *jet/accuracy_1_of_4‚0.9743425250053406 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9221220016479492 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6381868124008179 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6930125951766968 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22119168937206268 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6723760366439819 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6708695292472839 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9685583114624023ÒóßKß%‚(:À  •ôŠÓÔªé³ network.recv‚ 1.078224e+06 disk.drbd1550.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 2916.69921875# proc.memory.rssMB‚ 2937.07421875* proc.memory.percent‚11.950985590616863 disk.sda.in‚0.03125 disk.sda.out‚ 409.69921875 disk.drbd2041.out‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚47054.7744140625 disk.drbd1584.out‚0 disk.drbd2078.out‚0 network.sent‚ 1.5272498e+07 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 409.69921875 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0" disk.dm-0.out‚47054.7744140625 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1356.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.51379897315206$ disk./.usageGB‚1260.858341217041 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.03125 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1584.in‚0 disk.drbd2111.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 82.416015625 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1226.out‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 cpu‚25.332854348089597 disk.drbd1289.in‚0$ memory_percent‚88.13192049662271ÚY)ûžà%‚(:“ •ôŠÓìŽÊ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚288.473' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.15766666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞǘGá%‚(:¼  ¤ôŠÓéú³ disk.sda2.in‚0.03125 disk.sda2.out‚ 547.62109375 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 cpu‚25.332759869882018 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 101.6484375" disk.dm-0.out‚47055.6533203125 disk.drbd2041.out‚0 disk.rbd1.in‚0$ memory_percent‚88.13095092773438 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0% proc.memory.availableMB‚ 2916.9375* proc.memory.percent‚11.950985590616863% disk./.usageGB‚1260.8586158752441 disk.drbd1673.out‚0 disk.drbd1552.in‚0 network.recv‚ 1.08079e+06( disk./.usagePercent‚70.5138143334766 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.sda.in‚0.03125 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 547.62109375% disk.nvme0n1.out‚47055.6533203125 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 82.4482421875# proc.memory.rssMB‚ 2937.07421875 disk.drbd1356.out‚0 network.sent‚ 1.5326266e+07 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0>P[Æžâ%‚(:“ ¤ôŠÓÉÐí³ gpu.0.gpu‚95.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚212.839' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.94633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eS•4ã%/ _wandbruntime‚15850  _runtime‚15850”©„_Êä%Ä ' lr-AdamW/pg2‚3.8099454783073216e-05 trainer/global_step‚8699 % _timestamp‚1.7848550871139586e+09 _runtime‚15850.471180122 _step‚723 ' lr-AdamW/pg1‚3.8099454783073216e-05ºÑsÌÏå%É % _timestamp‚1.7848550871139586e+09 _runtime‚15850.471180122 _step‚723 ' lr-AdamW/pg1‚3.8099454783073216e-05 ' lr-AdamW/pg2‚3.8099454783073216e-05 trainer/global_step‚8699Ó8bXÁ4æ%/ _wandbruntime‚15850  _runtime‚15850WÅRõ (ç%ƒP GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9066737786661 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26522649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚696518921242112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2504991260687472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚8699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9065393175375 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2505979863746560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9081943563264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49709009 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚724 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26522349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75569700 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2515061833483264 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2505979889920000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9080598127104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2514056628434879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49708704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9081943563264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2515060461873664 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2505979863746560 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2514057998474133 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75569706 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚696519330114560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2515061833483264 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2515060461873664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23186360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11107398208512 % _timestamp‚1.7848550871149392e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156368618 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2504991235259504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚685411931906048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75569706 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11107357427200[ÝTñ(è%ˆP YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26522649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚696518921242112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2504991260687472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚8699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9065393175375 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2505979863746560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9081943563264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49709009 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚724 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26522349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75569700 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2515061833483264 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2505979889920000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9080598127104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2514056628434879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49708704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9081943563264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2515060461873664 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2505979863746560 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2514057998474133 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75569706 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚696519330114560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2515061833483264 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2515060461873664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23186360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11107398208512 % _timestamp‚1.7848550871149392e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156368618 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2504991235259504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚685411931906048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75569706 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11107357427200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9066737786661 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156396151 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75578067 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80818078 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2515341765288960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49714320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75578067 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80808000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2506258773037056 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9081620635136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.;ÞlÝ™allocated‚49714624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2506258799210496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386067 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚685490150180864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9067758588413 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75578073 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2515340393672192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 % _timestamp‚1.7848550885857658e+09 _step‚725 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ¹Kg'ë%“N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49714320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75578067 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80808000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2506258773037056 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9081620635136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49714624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2506258799210496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386067 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚685490150180864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9067758588413 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75578073 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2515340393672192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 % _timestamp‚1.7848550885857658e+09 _step‚725 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9081620635136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11108676528128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156396151 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156386067 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2515341765288960 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156396151 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75578067 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80818078 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2515341765288960ÕŠ³Ë4ì%/ _wandbruntime‚15851  _runtime‚15851*è` í%† 0loss/FRt2/assignment_loss‚0.5696068725017743 2loss/SRqqt1/assignment_loss‚0.5761634238804574 1loss/SRqqt2/detection_loss‚0.5901902029916921 .loss/FBt1/detection_loss‚0.695111502222999 /loss/FBt2/detection_loss‚0.6286723393965999 % _timestamp‚1.7848550886258333e+09 _runtime‚15851.983057389 0loss/FRt1/assignment_loss‚1.6960208997083681 3loss/FBt1/assignment_loss‚0.0006917870009913964 /loss/FRt1/detection_loss‚0.5842756756918804 1loss/SRqqt1/detection_loss‚0.9754453312546816 3loss/SRqqt2/assignment_loss‚0.46632699328264043 %loss/total_loss‚8.209293084322502 epoch‚31 trainer/global_step‚8699 2loss/FBt2/assignment_loss‚0.013332067494137897 /loss/FRt2/detection_loss‚1.4134559888962797 _step‚726’Vqî%‹ epoch‚31 trainer/global_step‚8699 2loss/FBt2/assignment_loss‚0.013332067494137897 /loss/FRt2/detection_loss‚1.4134559888962797 _step‚726 0loss/FRt2/assignment_loss‚0.5696068725017743 2loss/SRqqt1/assignment_loss‚0.5761634238804574 1loss/SRqqt2/detection_loss‚0.5901902029916921 .loss/FBt1/detection_loss‚0.695111502222999 /loss/FBt2/detection_loss‚0.6286723393965999 % _timestamp‚1.7848550886258333e+09 _runtime‚15851.983057389 0loss/FRt1/assignment_loss‚1.6960208997083681 3loss/FBt1/assignment_loss‚0.0006917870009913964 /loss/FRt1/detection_loss‚0.5842756756918804 1loss/SRqqt1/detection_loss‚0.9754453312546816 3loss/SRqqt2/assignment_loss‚0.46632699328264043 %loss/total_loss‚8.209293084322502ÖCuü×Nï%‚(:à  ³ôŠÓÁíâ³ disk.sda.out‚ 550.01171875$ memory_percent‚88.13112576802573 disk.drbd1289.out‚0 disk.drbd1938.out‚0 network.sent‚ 1.5329306e+07 disk.drbd2041.in‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2937.07421875* proc.memory.percent‚11.950985590616863 disk.nvme0n1.in‚ 101.6484375$ disk./.usageGB‚1260.857997894287 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0.03125 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 82.5498046875 disk.sda.in‚0.03125 disk.drbd1673.in‚0 disk.sda2.out‚ 550.01171875# disk.dm-0.out‚47056.78271484375 disk.drbd1828.in‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚47056.78271484375 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 cpu‚25.332713811716722 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 2916.89453125 disk.drbd1226.out‚0 disk.drbd2118.out‚0 network.recv‚ 1.081764e+06 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51377977274639 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0èë9žð%‚(:“ ³ôŠÓ‰ÙÁ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚272.006' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.66866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GŠÀNñ%‚(:à  ÂôŠÓѪڳ disk.sda2.out‚ 617.15625( proc.memory.availableMB‚ 2916.3984375 proc.cpu.threads‚17 disk.dm-0.in‚ 101.6484375 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 network.recv‚ 1.084434e+06 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 617.15625$ memory_percent‚88.13314437866211 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 network.sent‚ 1.5389322e+07 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.950985590616863% disk./.usageGB‚1260.8589630126953 disk.drbd1226.out‚0 disk.rbd1.out‚ 82.6123046875 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚ 0.03515625 cpu‚25.332651739873448 disk.drbd1938.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚47057.01416015625& disk.nvme0n1.out‚47057.01416015625) disk./.usagePercent‚70.51383374722012 disk.drbd2078.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1817.out‚0ÜÇ䊞ò%‚(:“ ÂôŠÓãǶ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚310.33' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.44333333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#e¶žó%‚(:“ ÑôŠÓ­Ê˳ gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.712' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.23733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô.Tô%‚(:É  ÑôŠÓ³ºè³ disk.rbd1.out‚ 82.6123046875 cpu‚25.332604907147388 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 101.6484375) disk./.usagePercent‚70.51383374722012 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚47058.28662109375 disk.drbd1289.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8589630126953# disk.dm-0.out‚47058.28662109375 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.sda1.in‚0 network.sent‚ 1.5392326e+07 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0( proc.memory.availableMB‚ 2916.3203125* proc.memory.percent‚11.950985590616863 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0$ memory_percent‚88.13346227010092 disk.drbd2118.in‚0 disk.sda.out‚ 730.21484375 disk.sda2.out‚ 730.21484375 disk.drbd1289.out‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.dm-0.in‚ 101.6484375 network.recv‚ 1.085397e+06 disk.drbd1584.in‚0_þžžõ%‚(:“ àôŠÓ©¦Ø³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚283.951' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.65033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±>ÓOö%‚(:Ä  àôŠÓŒîû³ disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 network.recv‚ 1.086345e+06 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 82.64453125 disk.sda1.out‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd2078.in‚0( proc.memory.availableMB‚ 2916.3203125& disk.nvme0n1.out‚47058.48974609375 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.950985590616863 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 730.5078125) disk./.usagePercent‚70.51383374722012 disk.sda1.in‚0% disk./.usageGB‚1260.8589630126953 disk.sda.in‚ 0.03515625$ memory_percent‚88.13346227010092 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1817.out‚0 cpu‚25.33255801138948 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 730.5078125 disk.drbd1931.out‚0 disk.drbd2078.out‚0 network.sent‚ 1.5395373e+07 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚47058.48974609375H$¼…‘÷%‚(:† ïôŠÓ¦Äϳ gpu.0.gpu‚37.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.82' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.94 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÏAuKø%‚(:À  ïôŠÓÑŒï³ disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.950985590616863 proc.cpu.threads‚17 network.recv‚ 1.087314e+06 disk.sda.out‚730.8125 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625# disk.dm-0.out‚47058.76806640625 disk.drbd2041.out‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.8589782714844 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 101.6484375) proc.memory.availableMB‚ 2916.31640625 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2118.in‚0$ memory_percent‚88.13347816467285) disk./.usagePercent‚70.51383460057149 disk.drbd1965.in‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 82.73828125 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2937.07421875 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 cpu‚25.332512044595944 network.sent‚ 1.5398388e+07 disk.drbd1673.out‚0 disk.sda2.out‚730.8125 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 101.6484375& disk.nvme0n1.out‚47058.76806640625 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda1.in‚0à›ãžù%‚(:“ þôŠÓÖ±Þ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.043' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.34766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å#èºRú%‚(:Ç  þôŠÓÌûø³ disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 82.80078125 disk.drbd1552.in‚0) disk./.usagePercent‚70.51383460057149 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 network.recv‚ 1.08838e+06 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0# disk.dm-0.out‚47058.97900390625$ memory_percent‚88.13347816467285 disk.drbd2118.in‚0 disk.sda2.out‚ 731.08984375 cpu‚25.332434262460342 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚ 101.6484375& disk.nvme0n1.out‚47058.97900390625 disk.drbd1510.out‚0) proc.memory.availableMB‚ 2916.31640625 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.sda.out‚ 731.08984375 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1584.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 network.sent‚ 1.5401393e+07% disk./.usageGB‚1260.8589782714844 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.950985590616863]¿P.4û%/ _wandbruntime‚15937  _runtime‚15937 Ÿz%Êü%Ä ' lr-AdamW/pg2‚3.7472577907149194e-05 trainer/global_step‚8749 % _timestamp‚1.7848551740810566e+09 _runtime‚15937.438310835 _step‚727 ' lr-AdamW/pg1‚3.7472577907149194e-05:íœØÏý%É ' lr-AdamW/pg2‚3.7472577907149194e-05 trainer/global_step‚8749 % _timestamp‚1.7848551740810566e+09 _runtime‚15937.438310835 _step‚727 ' lr-AdamW/pg1‚3.7472577907149194e-05׎wz4þ%/ _wandbruntime‚15937  _runtime‚15937•*eæßÿ%ƒP XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23318360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2529058487564288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157252990 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81254870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚8749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9133133119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2518931349376272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9131787683328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81254870 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2528049191045741 MGDeviceStatsMonitor.on_train_bNC®*atch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75988050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2528047821006487 $ _timestamp‚1.784855174082206e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49988689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2529057115954688 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157242920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81264934 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚689341250203648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9116497058183 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11170269988352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75988056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26670329 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚728Û|-S(€&ˆP MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75988050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2528047821006487 $ _timestamp‚1.784855174082206e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49988689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2529057115954688 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157242920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81264934 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚689341250203648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9116497058183 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11170269988352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75988056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26670329 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 _step‚728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23318360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2529058487564288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157252990 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81254870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚8749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9133133119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2518931349376272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9131787683328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81254870 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2528049191045741ØëfZd4&/ _wandbruntime‚15938  _runtime‚15938C|iA'‚&ŽN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157270453 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2520204263735296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157260369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9117517859823 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81263952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23321000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚689419836569600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49993875 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9132810191360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75996417 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2520204237561856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26673231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75996423 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75996417 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11171473264640 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11171528718848 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2528327643581903 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2529337047753216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49994231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚351 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9132810191360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚729 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2520204237561856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157270453 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 trainer/global_step‚8749 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2528329013621269 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26672880 % _timestamp‚1.7848551755598032e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157260369 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2519210125722080Ž ϵ]ƒ&“N XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2528329013621269 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26672880 % _timestamp‚1.7848551755598032e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157260369 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2519210125722080 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157270453 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2520204263735296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157260369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 PADeviceStatsMonitor.on_traimZ0ˆ¼n_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9117517859823 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81263952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23321000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚689419836569600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49993875 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9132810191360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75996417 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2520204237561856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26673231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75996423 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75996417 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11171473264640 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11171528718848 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2528327643581903 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2529337047753216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49994231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚351 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9132810191360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚729 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2520204237561856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157270453 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 trainer/global_step‚8749Ù¶W}04„&/ _wandbruntime‚15938  _runtime‚15938 ¦c …&† %loss/total_loss‚7.823441162685436 epoch‚31 0loss/FRt1/assignment_loss‚1.6027649639868269 /loss/FRt1/detection_loss‚0.5144882676494205 0loss/FBt2/detection_loss‚0.42362089363298006 % _timestamp‚1.7848551755969603e+09 _step‚730 2loss/FBt1/assignment_loss‚0.002479396886972296 1loss/FBt2/assignment_loss‚0.00557431865123731 trainer/global_step‚8749 2loss/SRqqt1/assignment_loss‚0.5121458108181671 3loss/SRqqt2/assignment_loss‚0.45967513034111357 /loss/FRt2/detection_loss‚1.4668229283542735 1loss/SRqqt1/detection_loss‚0.8835709224144097 /loss/FBt1/detection_loss‚0.6657219237086418 0loss/FRt2/assignment_loss‚0.7071132952932545 1loss/SRqqt2/detection_loss‚0.5794633109481382 _runtime‚15938.954177503rX†&‹ epoch‚31 0loss/FRt1/assignment_loss‚1.6027649639868269 /loss/FRt1/detection_loss‚0.5144882676494205 0loss/FBt2/detection_loss‚0.42362089363298006 % _timestamp‚1.7848551755969603e+09 _step‚730 2loss/FBt1/assignment_loss‚0.002479396886972296 1loss/FBt2/assignment_loss‚0.00557431865123731 trainer/global_step‚8749 2loss/SRqqt1/assignment_loss‚0.5121458108181671 3loss/SRqqt2/assignment_loss‚0.45967513034111357 /loss/FRt2/detection_loss‚1.4668229283542735 1loss/SRqqt1/detection_loss‚0.8835709224144097 /loss/FBt1/detection_loss‚0.6657219237086418 0loss/FRt2/assignment_loss‚0.7071132952932545 1loss/SRqqt2/detection_loss‚0.5794633109481382 _runtime‚15938.954177503 %loss/total_loss‚7.823441162685436Ú,³´)ž‡&‚(:“ õŠÓ«È©³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚208.383' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚69.46100000000001 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!„ÅÈKˆ&‚(:À  õŠÓ•èƳ disk.drbd2078.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.sda2.out‚731.4375) disk./.usagePercent‚70.51383460057149 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1931.in‚0 network.sent‚ 1.5404424e+07 disk.drbd1285.out‚0) proc.memory.availableMB‚ 2916.25390625 network.recv‚ 1.089453e+06 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0$ memory_percent‚88.13373247782388% disk./.usageGB‚1260.8589782714844 disk.dm-0.in‚ 101.6484375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0# disk.dm-0.out‚47059.36865234375 disk.drbd2111.out‚0& disk.nvme0n1.out‚47059.36865234375 disk.drbd1226.in‚0 disk.drbd1931.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 82.80078125 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda.out‚731.4375 disk.drbd1289.in‚0* proc.memory.percent‚11.950985590616863 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1965.in‚0 disk.drbd1673.out‚0 cpu‚25.332373822472174 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0{X€hž‰&‚(:“ œõŠÓ¯Þʳ gpu.0.gpu‚42.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.951' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.65033333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²¿ŸUŠ&‚(:Ê  œõŠÓØñã³ disk.drbd1226.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 2916.25390625 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚47059.58349609375 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1510.in‚0% disk./.usageGB‚1260.8570861816406 disk.drbd1584.out‚0 disk.drbd1817.in‚0 network.recv‚ 1.092106e+06) disk./.usagePercent‚70.51372878500243 disk.drbd1226.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚47059.58349609375 disk.drbd2118.out‚0$ memory_percent‚88.13373247782388 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.rbd1.out‚ 82.8330078125 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.950985590616863 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1285.out‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2937.07421875 network.sent‚ 1.5464995e+07 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 731.83203125 cpu‚25.332327083007517 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.sda.out‚ 731.83203125 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17Äó=Ÿ‹&‚(:” «õŠÓ›ãܳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.794' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.26466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02¹®SŒ&‚(:È  «õŠÓ˜Óü³ disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0# disk.dm-0.out‚47059.86669921875 disk.rbd1.out‚ 83.1064453125 disk.drbd2118.out‚0 disk.drbd1828.out‚0 network.sent‚ 1.5467455e+07 disk.drbd2118.in‚0 disk.sda2.out‚ 732.1328125 disk.drbd2041.out‚0 disk.drbd1285.in‚0$ memory_percent‚88.13373247782388 proc.cpu.threads‚17 disk.nvme0n1.in‚ 101.6484375 disk.drbd1931.in‚0 disk.drbd1356.out‚0 cpu‚25.332233668565852 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.drbd1817.in‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.950985590616863 disk.drbd1584.out‚0 disk.sda.out‚ 732.1328125 disk.drbd1673.out‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8570861816406 disk.drbd1552.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 2916.25390625 disk.drbd1550.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51372878500243& disk.nvme0n1.out‚47059.86669921875 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1828.in‚0 disk.rbd1.in‚0 network.recv‚ 1.093182e+06 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0¿ž\T&‚(:É  ºõŠÓˆ´ network.recv‚ 1.094311e+06 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 cpu‚25.33218760075455 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 732.53515625 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0$ memory_percent‚88.13373247782388 disk.drbd1673.out‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚47060.00341796875 disk.drbd1550.in‚0* proc.memory.percent‚11.950985590616863 network.sent‚ 1.5470512e+07 disk.drbd1356.out‚0) disk./.usagePercent‚70.51363150294699# proc.memory.rssMB‚ 2937.07421875% disk./.usageGB‚1260.8553466796875 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 732.53515625 disk.drbd1938.out‚0# disk.dm-0.out‚47060.00341796875 disk.rbd1.out‚ 83.1689453125 disk.drbd1285.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 2916.25390625 proc.cpu.threads‚17G‘yËžŽ&‚(:“ ºõŠÓåÃè³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚263.351' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.78366666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É=T&‚(:É  ÉõŠÓƹ÷³ disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.rbd1.out‚ 83.1689453125# proc.memory.rssMB‚ 2937.07421875* proc.memory.percent‚11.950985590616863 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.sda2.out‚ 732.81640625 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚47060.35986328125 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0$ memory_percent‚88.13373247782388 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625# disk.dm-0.out‚47060.35986328125 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8553466796875 disk.sda1.in‚0) proc.memory.availableMB‚ 2916.25390625 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 network.sent‚ 1.5473625e+07 network.recv‚ 1.09531e+06 disk.drbd1552.in‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.51363150294699 disk.nvme0n1.in‚ 101.6484375 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 101.6484375 disk.sda.out‚ 732.81640625 disk.drbd1965.out‚0 cpu‚25.332157934940078 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1584.out‚0åÒêkœ&‚(:‘ ÉõŠÓýÆܳ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.037' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.01233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D o‡O‘&‚(:Ä  ØõŠÓ‘æý³ disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0$ memory_percent‚85.10947227478027 network.recv‚ 1.09632e+06 disk.drbd2111.out‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.6484375 disk.drbd1356.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51366094356904 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 733.203125 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0# disk.dm-0.out‚47060.89501953125 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 101.6484375 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 733.203125 disk.drbd1931.out‚0 network.sent‚ 1.5476678e+07 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.950985590616863 cpu‚25.332096575009352 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2937.07421875% disk./.usageGB‚1260.8558731079102 disk.sda1.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚47060.89501953125 disk.rbd1.out‚ 83.201171875) proc.memory.availableMB‚ 3659.49609375 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0ùˆ•{ž’&‚(:“ ØõŠÓ¸Õá³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.042' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.34733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P'4“&/ _wandbruntime‚16023  _runtime‚16023b4¡¬È”& & lr-AdamW/pg1‚3.684846570048162e-05 & lr-AdamW/pg2‚3.684846570048162e-05 trainer/global_step‚8799 % _timestamp‚1.7848552606307511e+09 _runtime‚16023.988845738 _step‚731¯ ªÍ•&Ç & lr-AdamW/pg1‚3.684846570048162e-05 & lr-AdamW/pg2‚3.684846570048162e-05 trainer/global_step‚8799 % _timestamp‚1.7848552606307511e+09 _runtime‚16023.988845738 _step‚731Û $8s4–&/ _wandbruntime‚16023  _runtime‚16023A§ð (—&„P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81720886 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2543055141645312 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2542039013578095 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2533870792796160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9182977239552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9184322675712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23450360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2533870792796160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚704503444439040 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚16023.989088137 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50267561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50267923 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2532871438065072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2533870818969600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81710822 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2543055141645312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158127292 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81720886 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81710822 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚362 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2532871412637104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9168945552277 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76406400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9167600940991 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2542040383617349 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 % _timestamp‚1.7848552606315496e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182977239552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11232875937792 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2543053770035712 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2533870818969600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2543053770035712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76406400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚704503020886528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693270568501248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76406406 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26817206 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2532871438065072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2533870818969600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81710822 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2543055141645312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158127292 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81720886 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81710822 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚362 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2532871412637104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9168945552277 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76406400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9167600940991 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2542040383617349 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 % _timestamp‚1.7848552606315496e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182977239552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11232875937792 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2543053770035712 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2533870818969600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2543053770035712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76406400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064Ü×4™&/ _wandbruntime‚16025  _runtime‚16025æ±%¨'š&ŽN T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2543333701834240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81729982 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81729982 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23452995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76414767 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158144755 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2534149702086656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23453000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2543335073451008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81719904 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9168621742631 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76414767 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2534149702086656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26820105 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2542320206192877 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2534149728260096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚351 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11234127777792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158144755 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9185345190912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704583282644992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693349154867200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693348786776064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11234078615040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2533150214410880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158134671 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9183999747584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81719904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158134671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26820456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚704582865391104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚8799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50273456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2534149728260096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9169966354029 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50273100 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848552621090035e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183999747584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0d-u»'›&“N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚16025.466589141 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2542318836153511 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2543335073451008 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158134671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26820456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚704582865391104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚8799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50273456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2534149728260096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9169966354029 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50273100 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 % _timestamp‚1.7848552621090035e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183999747584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2543333701834240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81729982 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81729982 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23452995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76414767 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158144755 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2534149702086656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23453000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2543335073451008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81719904 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9168621742631 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76414767 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2534149702086656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26820105 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2542320206192877 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2534149728260096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚351 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11234127777792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158144755 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9185345190912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704583282644992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693349154867200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693348786776064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11234078615040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2533150214410880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158134671 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9183999747584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81719904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FŽT¦&‚(:É  ”öŠÓÓ…ì³ disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1550.in‚0 disk.drbd2041.in‚0 disk.drbd2118.in‚0 cpu‚25.331962002174233 disk.drbd1673.out‚0& disk.nvme0n1.out‚47062.26025390625 disk.drbd1550.out‚0) disk./.usagePercent‚70.51376803916514 disk.rbd1.out‚ 83.599609375 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 network.sent‚ 1.5545704e+07 disk.dm-0.in‚ 101.85546875 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2937.07421875 disk.drbd1289.out‚0$ memory_percent‚89.84711964925131% disk./.usageGB‚1260.8577880859375 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2111.in‚0 network.recv‚ 1.102179e+06 disk.sda2.in‚ 0.03515625 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.950985590616863 disk.sda.out‚ 734.67578125 disk.sda2.out‚ 734.67578125 disk.drbd1931.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚47062.26025390625 disk.drbd1965.in‚0' proc.memory.availableMB‚ 2495.171875 disk.drbd2111.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17²v2V§&‚(:Ë  £öŠÓ¹©´ disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 cpu‚25.331885236025187 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.sda.out‚ 734.93359375 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 83.693359375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0$ memory_percent‚91.23377799987793) proc.memory.availableMB‚ 2154.38671875 network.recv‚ 1.103046e+06& disk.nvme0n1.out‚47062.56787109375 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2937.07421875% disk./.usageGB‚1260.8577880859375 disk.drbd1584.in‚0# disk.dm-0.out‚47062.56787109375 disk.drbd2078.out‚0) disk./.usagePercent‚70.51376803916514 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625* proc.memory.percent‚11.950985590616863 network.sent‚ 1.5548713e+07 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 734.93359375 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0z78¨&‚(:’ £öŠÓÙ…æ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.19' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.06333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶x#“©&‚(:ˆ ²öŠÓ”ûà³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚246.582' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚82.194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F2yUª&‚(:Ê  ²öŠÓÉéú³ disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 735.24609375 disk.drbd1552.in‚0 disk.drbd1673.out‚0 network.recv‚ 1.103967e+06) disk./.usagePercent‚70.51376803916514 disk.drbd1938.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.950985590616863# disk.dm-0.out‚47062.81005859375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2937.07421875 cpu‚25.331840281978213& disk.nvme0n1.out‚47062.81005859375 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 83.755859375 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 network.sent‚ 1.5551766e+07% disk./.usageGB‚1260.8577880859375 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1944.1640625 disk.sda2.in‚ 0.03515625$ memory_percent‚92.08917617797852 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 735.24609375 disk.drbd2078.in‚0¹9²”4«&/ _wandbruntime‚16110  _runtime‚16110€ìÂÔɬ&à ' lr-AdamW/pg1‚3.6227199969881654e-05 ' lr-AdamW/pg2‚3.6227199969881654e-05 trainer/global_step‚8849 $ _timestamp‚1.784855347561004e+09 _runtime‚16110.918291268 _step‚735Œ>åέ&È ' lr-AdamW/pg1‚3.6227199969881654e-05 ' lr-AdamW/pg2‚3.6227199969881654e-05 trainer/global_step‚8849 $ _timestamp‚1.784855347561004e+09 _runtime‚16110.918291268 _step‚735ßä•À4®&/ _wandbruntime‚16110  _runtime‚16110¢´Àr (¯&ƒP TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82166774 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50546919 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚697199886798848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11295423713792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2546811526753872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2546811501325904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23582355 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2557050424116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚371 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚8849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76824750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50546548 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76824750 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11295366155264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2556031576188957 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚708494884862976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76824756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2547816257320960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9218704823799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82176838 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26964559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2547816257320960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9234166795776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159001594 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2557050424116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2557051795726336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2547816283494400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2556030206149703 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23582360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚366 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚697199518707712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9235512231936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2547816283494400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76824756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26964193 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9220049435085 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9235512231936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158991524 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _runtime‚16110.919669118 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2557051795726336 $ _timestamp‚1.784855347562127e+09äõáù(°&ˆP \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82176838 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26964559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2547816257320960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9234166795776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159001594 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2557050424116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2557051795726336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2547816283494400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2556030206149703 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23582360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚366 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚697199518707712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9235512231936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2547816283494400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76824756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26964193 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9220049435085 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9235512231936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158991524 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _runtime‚16110.919669118 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2557051795726336 $ _timestamp‚1.784855347562127e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82166774 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50546919 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚697199886798848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11295423713792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2546811526753872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2546811501325904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23582355 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2557050424116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚371 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚8849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76824750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50546548 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76824750 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11295366155264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2556031576188957 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚708494884862976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76824756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2547816257320960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9218704823799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136à©:Â#4±&/ _wandbruntime‚16112  _runtime‚16112úç;U ²&ŽN TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2547090328527648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76833117 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚708575153162752 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2547090303099680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9236534747136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2557331727532032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50552111 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159019057 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82175856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9221070236837 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9219725625439 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76833123 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9236534747136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82185934 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2556310028725119 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2548095166611456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159019057 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2556311398764485 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82185934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26967116 _runtime‚16112.395113228 _step‚737 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2548095192784896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26967473 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848553490374656e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2548095166611456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159019057 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2556311398764485 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82185934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26967116 _runtime‚16112.395113228 _step‚737 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2548095192784896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26967473 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848553490374656e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76833117 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚708575153162752 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2547090303099680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9236534747136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2557331727532032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50552111 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159019057 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82175856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9221070236837 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9219725625439 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824áô åw4´&/ _wandbruntime‚16112  _runtime‚16112¦šéµ&þ 0loss/FBt2/detection_loss‚0.43616814279615446 _runtime‚16112.433241819 _step‚738 3loss/SRqqt2/assignment_loss‚0.49081269740233785 /loss/FBt1/detection_loss‚0.7075763849631673 2loss/SRqqt1/assignment_loss‚0.5404978324335017 2loss/FBt2/assignment_loss‚0.053432630322131545 epoch‚31 trainer/global_step‚8849 /loss/FRt1/assignment_loss‚1.596703728883429 0loss/FRt2/assignment_loss‚0.7039875779795436 /loss/FRt2/detection_loss‚1.4855670352426227 0loss/SRqqt1/detection_loss‚0.959031289328598 1loss/SRqqt2/detection_loss‚0.5381693268569234 $loss/total_loss‚8.03601870490757 ! _timestamp‚1.784855349076e+09 1loss/FBt1/assignment_loss‚0.00414528597854284 .loss/FRt1/detection_loss‚0.519926772720618CˆJ ¶&ƒ 2loss/SRqqt1/assignment_loss‚0.5404978324335017 2loss/FBt2/assignment_loss‚0.053432630322131545 epoch‚31 trainer/global_step‚8849 /loss/FRt1/assignment_loss‚1.596703728883429 0loss/FRt2/assignment_loss‚0.7039875779795436 /loss/FRt2/detection_loss‚1.4855670352426227 0loss/SRqqt1/detection_loss‚0.959031289328598 1loss/SRqqt2/detection_loss‚0.5381693268569234 $loss/total_loss‚8.03601870490757 ! _timestamp‚1.784855349076e+09 1loss/FBt1/assignment_loss‚0.00414528597854284 .loss/FRt1/detection_loss‚0.519926772720618 0loss/FBt2/detection_loss‚0.43616814279615446 _runtime‚16112.433241819 _step‚738 3loss/SRqqt2/assignment_loss‚0.49081269740233785 /loss/FBt1/detection_loss‚0.7075763849631673âûGÛ#U·&‚(:Ê  ÁöŠÓÓ•´% disk./.usageGB‚1260.8577880859375 disk.drbd1965.in‚0& disk.nvme0n1.out‚47063.16455078125 disk.drbd1552.out‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51376803916514 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 83.755859375$ memory_percent‚92.08644231160483( proc.memory.availableMB‚ 1944.8359375# disk.dm-0.out‚47063.16455078125 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 cpu‚25.331779632210264 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 735.93359375 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0 disk.sda.out‚ 735.93359375 network.sent‚ 1.5554783e+07 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2937.07421875* proc.memory.percent‚11.950985590616863 network.recv‚ 1.105012e+06 disk.drbd1931.out‚0Ãônž¸&‚(:“ ÁöŠÓ«þá³ gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.833' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.94433333333335 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~*Šž¹&‚(:“ ÐöŠÓ•Ø·³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚272.518' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.83933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æȇDNº&‚(:à  ÐöŠÓ”Ú³ disk.dm-0.in‚ 101.85546875 network.sent‚ 1.5614952e+07 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51376931919219 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17* proc.memory.percent‚11.950985590616863 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1552.out‚0 disk.sda2.out‚ 736.2578125 disk.drbd1965.in‚0$ disk.nvme0n1.out‚47063.470703125 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 736.2578125 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0' proc.memory.availableMB‚ 1945.078125 disk.drbd2118.in‚0 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2937.07421875! disk.dm-0.out‚47063.470703125 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 cpu‚25.331720228709266$ disk./.usageGB‚1260.857810974121 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 101.85546875 network.recv‚ 1.107715e+06 disk.drbd1673.in‚0 disk.rbd1.out‚ 83.7880859375$ memory_percent‚92.08545684814453 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1938.in‚0çôcK‘»&‚(:† ßöŠÓÏçó gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚211.08' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚70.36 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ììþV¼&‚(:Ë  ßöŠÓœŒã³) disk./.usagePercent‚70.51377102589491 disk.drbd1673.out‚0 disk.sda2.out‚ 736.78515625 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 cpu‚25.33165972175247 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚ 0.03515625 network.sent‚ 1.5618557e+07 disk.drbd1938.in‚0# disk.dm-0.out‚47064.01220703125 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1944.83203125 disk.drbd1356.out‚0 disk.drbd1828.out‚0 network.recv‚ 1.108765e+06 disk.dm-0.in‚ 101.85546875 disk.drbd1965.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.rbd1.out‚ 83.8818359375 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2937.07421875 proc.cpu.threads‚17 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.950985590616863 disk.drbd1289.out‚0 disk.drbd2111.in‚0% disk./.usageGB‚1260.8578414916992 disk.sda1.in‚0$ memory_percent‚92.08645820617676 disk.sda.out‚ 736.78515625 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚47064.01220703125 disk.drbd1510.out‚0P¡6Ÿ½&‚(:” îöŠÓܱç³ gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.115' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.038333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q‡E7R¾&‚(:Ç  îöŠÓ‡á†´ disk.rbd1.in‚0% disk.nvme0n1.out‚47064.3154296875 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 737.04296875 cpu‚25.33158322002749 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.950985590616863 disk.drbd2078.in‚0 disk.drbd1931.out‚0" disk.dm-0.out‚47064.3154296875' proc.memory.availableMB‚ 1944.828125# proc.memory.rssMB‚ 2937.07421875 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd1552.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0% disk./.usageGB‚1260.8578033447266 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 network.sent‚ 1.5621019e+07 disk.sda.out‚ 737.04296875 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 83.9443359375 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51376889251651 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.dm-0.in‚ 101.85546875$ memory_percent‚92.08647410074869 network.recv‚ 1.109775e+06 disk.sda.in‚ 0.03515625 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0 proc.cpu.threads‚17©3ýQž¿&‚(:“ ýöŠÓ®Î±³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚272.693' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.89766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼_š†UÀ&‚(:Ê  ýöŠÓ®Ï³ disk.drbd1552.in‚0 disk.drbd1965.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.drbd1828.in‚0& disk.nvme0n1.out‚47064.90966796875 disk.sda.in‚ 0.03515625# disk.dm-0.out‚47064.90966796875 disk.drbd2041.in‚0 network.sent‚ 1.5624115e+07 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1552.out‚0 network.recv‚ 1.110863e+06% disk./.usageGB‚1260.8576622009277 disk.sda.out‚ 737.30859375 disk.sda2.in‚ 0.03515625 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1944.83203125 disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 83.9443359375 disk.drbd1226.in‚0* proc.memory.percent‚11.950985590616863( disk./.usagePercent‚70.5137609990164 disk.sda2.out‚ 737.30859375 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.08645820617676 disk.drbd2078.out‚0 disk.rbd1.in‚0 cpu‚25.33154001829995 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 101.85546875êàÝÆ4Á&/ _wandbruntime‚16197  _runtime‚16197V¿uûÇÂ&Á $ _timestamp‚1.784855434517876e+09 _runtime‚16197.875122029 _step‚739 & lr-AdamW/pg1‚3.560886214905268e-05 & lr-AdamW/pg2‚3.560886214905268e-05 trainer/global_step‚8899úKµžÌÃ&Æ _runtime‚16197.875122029 _step‚739 & lr-AdamW/pg1‚3.560886214905268e-05 & lr-AdamW/pg2‚3.560886214905268e-05 trainer/global_step‚8899 $ _timestamp‚1.784855434517876e+09ã·Ñú’4Ä&/ _wandbruntime‚16197  _runtime‚16197«ë׶ Å&„P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159875896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚8899 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2561761748019200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2560751615442672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9286701788160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23714360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11358189540864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9286701788160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚350 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2561761721845760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50825909 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚701129205096448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚345 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82632790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27111549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27111204 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9269808706607 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2561761748019200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50825559 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚701128837005312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2571047078197760 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9271153317893 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚16197.876534328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2571048449807360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7848554345189538e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82632790 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2571047078197760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9285356352000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2561761721845760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚712486981570560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11358144565248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 t³‚(Æ&‰P [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9286701788160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚350 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2561761721845760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50825909 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚701129205096448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚345 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82632790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27111549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27111204 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9269808706607 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2561761748019200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50825559 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚701128837005312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2571047078197760 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9271153317893 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _runtime‚16197.876534328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2571048449807360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7848554345189538e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82632790 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2571047078197760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9285356352000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2561761721845760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚712486981570560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11358144565248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159875896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚8899 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2561761748019200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2560751615442672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9286701788160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23714360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11358189540864äôâ¿#4Ç&/ _wandbruntime‚16199  _runtime‚16199!=0å'È&ŽN H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159883275 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77251467 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _step‚741 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23716995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77251473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50831112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77251473 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2571328381613056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚701207791462400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11359444136448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848554359990973e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159883275 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159893359 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82641886 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2570302591336093 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2571328381613056 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2562040657309696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27114454 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚337 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712567235598848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9287724303360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82631808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9270829508247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9286378860032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9286378860032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2561030417216448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77251467 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23717000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82631808 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9287724303360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27114117 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112‘-ÁË2É&“N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77251473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50831112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77251473 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2571328381613056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚701207791462400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11359444136448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848554359990973e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159883275 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159893359 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82641886 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2570302591336093 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2571328381613056 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2562040657309696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27114454 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚337 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712567235598848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9287724303360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82631808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9270829508247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9286378860032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9286378860032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2561030417216448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77251467 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23717000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82631808 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9287724303360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27114117 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159883275 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77251467 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _step‚741 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23716995åtPÚ4Ê&/ _wandbruntime‚16199  _runtime‚16199GRËð Ë&‡ 0loss/FRt1/assignment_loss‚1.6343594326499407 2loss/SRqqt2/assignment_loss‚0.4801752954764362 /loss/FRt2/detection_loss‚1.4704201617788006 0loss/FBt2/detection_loss‚0.43051136335760937 _runtime‚16199.389985365 _step‚742 0loss/FRt2/assignment_loss‚0.6246802676333129 3loss/SRqqt1/assignment_loss‚0.48466192583288764 3loss/FBt1/assignment_loss‚7.288072214396746e-05 epoch‚31 2loss/FBt2/assignment_loss‚0.025344262952898943 /loss/FRt1/detection_loss‚0.5281636243132234 1loss/SRqqt1/detection_loss‚0.8934950696928207 1loss/SRqqt2/detection_loss‚0.6335176697988211 trainer/global_step‚8899 /loss/FBt1/detection_loss‚0.4670659805678071 %loss/total_loss‚7.672467934776702 $ _timestamp‚1.784855436032712e+09I›£`Ì&Œ 0loss/FBt2/detection_loss‚0.43051136335760937 _runtime‚16199.389985365 _step‚742 0loss/FRt2/assignment_loss‚0.6246802676333129 3loss/SRqqt1/assignment_loss‚0.48466192583288764 3loss/FBt1/assignment_loss‚7.288072214396746e-05 epoch‚31 2loss/FBt2/assignment_loss‚0.025344262952898943 /loss/FRt1/detection_loss‚0.5281636243132234 1loss/SRqqt1/detection_loss‚0.8934950696928207 1loss/SRqqt2/detection_loss‚0.6335176697988211 trainer/global_step‚8899 /loss/FBt1/detection_loss‚0.4670659805678071 %loss/total_loss‚7.672467934776702 $ _timestamp‚1.784855436032712e+09 0loss/FRt1/assignment_loss‚1.6343594326499407 2loss/SRqqt2/assignment_loss‚0.4801752954764362 /loss/FRt2/detection_loss‚1.4704201617788006æ@³QÍ&‚(:Æ  Œ÷ŠÓÂÝ´ disk.sda.out‚ 737.76953125 disk.sda2.out‚ 737.76953125 disk.drbd1289.out‚0& disk.nvme0n1.out‚47065.46826171875 disk.drbd1226.out‚0 disk.drbd1285.out‚0 disk.dm-0.in‚ 101.85546875 network.recv‚ 1.11182e+06% disk./.usageGB‚1260.8576202392578 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚47065.46826171875 disk.rbd1.out‚ 83.9765625 proc.cpu.threads‚17$ memory_percent‚92.08672841389975 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.950985590616863 cpu‚25.331478792391415 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0' proc.memory.availableMB‚ 1944.765625 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2937.07421875 disk.sda1.in‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51375865230014 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 network.sent‚ 1.5627664e+07 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.rbd1.in‚0Ðyr¿ŸÎ&‚(:” Œ÷ŠÓý‹à³ gpu.0.gpu‚61.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.955' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.65166666666666 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êy-1IÏ&‚(:¾  ›÷ŠÓÂÜ׳ disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0$ proc.memory.availableMB‚1944.875 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0" disk.dm-0.out‚47066.0712890625 disk.drbd2078.in‚0% disk.nvme0n1.out‚47066.0712890625 disk.drbd1552.in‚0 disk.drbd1931.in‚0% disk./.usageGB‚1260.8576393127441 disk.drbd1938.out‚0 disk.drbd2041.out‚0$ memory_percent‚92.08628336588541 network.recv‚ 1.114747e+06 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.950985590616863 cpu‚25.331420026432166 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 84.0703125) disk./.usagePercent‚70.51375971898935 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1673.out‚0 disk.sda2.out‚ 737.984375 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 737.984375 disk.dm-0.in‚ 101.85546875 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2937.07421875 network.sent‚ 1.5687188e+07 disk.drbd1226.in‚0£ËHÐ&‚(:’ ›÷ŠÓÇÛ·³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.78' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.59333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êN8mžÑ&‚(:“ ª÷ŠÓÄݳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.251' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.75033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ÔôNÒ&‚(:à  ª÷ŠÓ…ý³ network.recv‚ 1.115769e+06) disk./.usagePercent‚70.51375865230014 disk.sda.out‚ 738.3046875 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 84.1328125# proc.memory.rssMB‚ 2937.07421875* proc.memory.percent‚11.950985590616863 disk.drbd2078.in‚0 disk.drbd1510.in‚0# memory_percent‚92.0876661936442 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 101.85546875" disk.dm-0.out‚47066.2705078125 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1550.out‚0 network.sent‚ 1.5690188e+07% disk./.usageGB‚1260.8576202392578 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 1944.53515625 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1510.out‚0 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 738.3046875 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚47066.2705078125 cpu‚25.33135937319403 disk.drbd2078.out‚0 disk.drbd1550.in‚0C½‹ŸÓ&‚(:” ¹÷ŠÓ§Ï³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚303.501' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.16699999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë¨û“KÔ&‚(:À  ¹÷ŠÓãÕä³ disk.drbd1938.out‚0% disk./.usageGB‚1260.8576202392578 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2937.07421875 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 738.63671875 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.rbd1.out‚84.21875 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0$ disk.nvme0n1.out‚47066.693359375 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1356.in‚0 network.sent‚ 1.5693181e+07 disk.drbd2118.in‚0 disk.drbd2041.out‚0 cpu‚25.33134643988635* proc.memory.percent‚11.950985590616863! disk.dm-0.out‚47066.693359375 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda.out‚ 738.63671875 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 101.85546875 proc.cpu.threads‚17) disk./.usagePercent‚70.51375865230014 disk.drbd1550.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.08736419677734 network.recv‚ 1.116753e+06 disk.sda.in‚ 0.03515625 disk.drbd1965.in‚0' proc.memory.availableMB‚ 1944.609375 õOÕ&‚(:Ä  È÷ŠÓÏë´ disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 739.00390625 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 739.00390625 disk.drbd2078.out‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 1944.2890625 disk.drbd1285.in‚0% disk./.usageGB‚1260.8576278686523 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.950985590616863 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.dm-0.in‚ 101.85546875 disk.drbd1289.in‚0 disk.drbd1965.out‚0$ memory_percent‚92.08866755167642 network.recv‚ 1.117757e+06 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚47066.91748046875 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚47066.91748046875 disk.rbd1.out‚84.28125 disk.drbd1285.out‚0 network.sent‚ 1.569618e+07 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2937.07421875 cpu‚25.33127080969704 disk.drbd1931.out‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51375907897582 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0Õ+òTžÖ&‚(:“ È÷ŠÓ—Þ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.233' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.74433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!èõ·ž×&‚(:“ ×÷ŠÓÍļ³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.468' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.48933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€â¶¥QØ&‚(:Æ  ×÷ŠÓÐÌÞ³ network.recv‚ 1.118848e+06 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 101.85546875% disk./.usageGB‚1260.8576278686523 disk.drbd1552.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2937.07421875 network.sent‚ 1.569922e+07 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 739.52734375 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.950985590616863 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 739.52734375 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0% disk.nvme0n1.out‚47067.3525390625 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0" disk.dm-0.out‚47067.3525390625 disk.rbd1.out‚ 84.4609375 disk.drbd1285.out‚0) disk./.usagePercent‚70.51375907897582 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0$ memory_percent‚92.08868344624838 cpu‚25.331212284560745 disk.drbd1673.in‚0) proc.memory.availableMB‚ 1944.28515625pcì¬4Ù&/ _wandbruntime‚16284  _runtime‚16284E¼rÉÚ&à ' lr-AdamW/pg1‚3.4993533287916175e-05 ' lr-AdamW/pg2‚3.4993533287916175e-05 trainer/global_step‚8949 % _timestamp‚1.7848555211684523e+09 _runtime‚16284.52569792 _step‚743[ªÀ ÎÛ&È _step‚743 ' lr-AdamW/pg1‚3.4993533287916175e-05 ' lr-AdamW/pg2‚3.4993533287916175e-05 trainer/global_step‚8949 % _timestamp‚1.7848555211684523e+09 _runtime‚16284.52569792ç-RËø4Ü&/ _wandbruntime‚16284  _runtime‚16284“Æ™ñ (Ý&„P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27258370 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2575707212544000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2585043732278784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77661450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83078678 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2574691704131472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9320912589415 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705058523394048 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2584013961332173 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9337891344384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9336545908224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51104730 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716479612455424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11421089061376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83088742 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77661456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23846360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 trainer/global_step‚8949 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2585045103888384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9322257200701 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9337891344384 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2585043732278784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848555211695971e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77661456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83088742 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27258021 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360{<Ï› Þ&‰P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848555211695971e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77661456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83088742 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27258021 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27258370 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2575707212544000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2585043732278784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77661450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83078678 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2574691704131472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9320912589415 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.v‚tfreed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705058523394048 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2584013961332173 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9337891344384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9336545908224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51104730 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716479612455424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11421089061376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83088742 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77661456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23846360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 trainer/global_step‚8949 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2585045103888384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9322257200701 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9337891344384 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2585043732278784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528èˆ 3w4ß&/ _wandbruntime‚16286  _runtime‚16286;ÆÏÎ'à&N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _runtime‚16286.006179868 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160757577 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2575986121834496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27260950 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11422303242752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9323278002453 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705137109760000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2574970505905248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705136741668864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83087760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83097838 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51110300 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77669823 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83087760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚716559044911616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2575986095661056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23848995 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160767661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27261300 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160757577 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2575986121834496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2574970480477280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9338913859584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚8949 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9321933391055 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚745 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716559460068352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51109945 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160767661 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77669817 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9337568416256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11422350308352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2584293783907701¦äŒg'á&’N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2575986121834496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27260950 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11422303242752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9323278002453 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705137109760000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2574970505905248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705136741668864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83087760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83097838 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51110300 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77669823 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83087760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚716559044911616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2575986095661056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23848995 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160767661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27261300 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160757577 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2575986121834496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2574970480477280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9338913859584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚8949 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9321933391055 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚745 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716559460068352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51109945 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160767661 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77669817 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9337568416256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11422350308352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2584293783907701 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _runtime‚16286.006179868 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160757577 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0éJú4â&/ _wandbruntime‚16286  _runtime‚16286éûßã&ˆ %loss/total_loss‚7.971566236295099 trainer/global_step‚8949 _runtime‚16286.040753507 /loss/FRt1/detection_loss‚0.5257237605112205 epoch‚31 _step‚746 2loss/FBt2/assignment_loss‚0.012814339600726175 1loss/SRqqt1/detection_loss‚0.9737769989955305 % _timestamp‚1.7848555226835265e+09 4loss/FBt1/assignment_loss‚0.00019343057152039952 /loss/FRt2/detection_loss‚1.4236948682336759 /loss/FBt1/detection_loss‚0.5346889892179105 0loss/FBt2/detection_loss‚0.48895827024029104 0loss/FRt1/assignment_loss‚1.6066833253658208 0loss/FRt2/assignment_loss‚0.6769753535588582 2loss/SRqqt1/assignment_loss‚0.6005914980831387 2loss/SRqqt2/assignment_loss‚0.5019913226138295 1loss/SRqqt2/detection_loss‚0.6254740793025765¥]·%ä& /loss/FRt2/detection_loss‚1.4236948682336759 /loss/FBt1/detection_loss‚0.5346889892179105 0loss/FBt2/detection_loss‚0.48895827024029104 0loss/FRt1/assignment_loss‚1.6066833253658208 0loss/FRt2/assignment_loss‚0.6769753535588582 2loss/SRqqt1/assignment_loss‚0.6005914980831387 2loss/SRqqt2/assignment_loss‚0.5019913226138295 1loss/SRqqt2/detection_loss‚0.6254740793025765 %loss/total_loss‚7.971566236295099 trainer/global_step‚8949 _runtime‚16286.040753507 /loss/FRt1/detection_loss‚0.5257237605112205 epoch‚31 _step‚746 2loss/FBt2/assignment_loss‚0.012814339600726175 1loss/SRqqt1/detection_loss‚0.9737769989955305 % _timestamp‚1.7848555226835265e+09 4loss/FBt1/assignment_loss‚0.00019343057152039952ê[ÑPå&‚(:Å  æ÷ŠÓ»ª´ disk.sda2.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0$ memory_percent‚92.08993911743164 cpu‚25.331152096022088( proc.memory.availableMB‚ 1943.9765625# proc.memory.rssMB‚ 2937.07421875 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51375801228662 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 network.recv‚ 1.119772e+06 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 84.5234375* proc.memory.percent‚11.950985590616863 network.sent‚ 1.5702219e+07 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 739.85546875% disk.nvme0n1.out‚47067.6611328125 disk.drbd2118.out‚0 disk.sda.out‚ 739.85546875 disk.drbd1673.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚47067.6611328125 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0$ disk./.usageGB‚1260.857608795166 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.dm-0.in‚ 101.85546875¨´zsžæ&‚(:“ æ÷ŠÓíáݳ gpu.0.gpu‚75.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.867' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.95566666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§·‘¶Sç&‚(:È  õ÷ŠÓù耴 disk.drbd1938.out‚0# disk.dm-0.out‚47067.99365234375 disk.drbd1965.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2939.37890625 disk.drbd1931.out‚0$ disk./.usageGB‚1260.857608795166 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51375801228662 disk.drbd1584.out‚0 disk.sda.out‚ 740.12890625 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1941.47265625 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 740.12890625 disk.drbd1550.in‚0 disk.drbd1226.in‚0$ memory_percent‚92.10012753804526 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 84.5234375 disk.drbd1550.out‚0 disk.dm-0.in‚ 101.85546875 proc.cpu.threads‚17 disk.drbd1356.in‚0* proc.memory.percent‚11.960363388061523 disk.drbd1285.in‚0 cpu‚25.331291817927244 network.recv‚ 1.122498e+06 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 network.sent‚ 1.5762244e+07 disk.drbd2111.out‚0& disk.nvme0n1.out‚47067.99365234375 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 101.85546875` ßJè&‚(:’ õ÷ŠÓÏ®üÏ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚177.836' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚59.278666666666666 gpu.0.smClock‚1410 gpu.0.memoryCï ‰,Slock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰LºÿSé&‚(:È  „øŠÓÍÃõµ disk.sda2.in‚ 0.03515625 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚47068.20849609375 network.recv‚ 1.123474e+06 disk.drbd1965.out‚0 disk.drbd2041.in‚0# memory_percent‚92.1156088511149) proc.memory.availableMB‚ 1937.66796875 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1356.out‚0 network.sent‚ 1.5768082e+07 disk.drbd1584.in‚0 disk.rbd1.out‚ 84.5556640625 cpu‚25.333458248419745 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.sda2.out‚ 740.4609375 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2943.33984375 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.976480484008789 disk.drbd1673.out‚0 disk.drbd1965.in‚0$ disk./.usageGB‚1260.857608795166 disk.drbd1356.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51375801228662 disk.drbd1552.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda.out‚ 740.4609375# disk.dm-0.out‚47068.20849609375 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0+xˆŠ”ê&‚(:‰ „øŠÓŽ’´ gpu.0.gpu‚96.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚316.026' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚105.342 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ý4ë&/ _wandbruntime‚16323  _runtime‚16323œ—¥ÿ:ì&ùu =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6367247700691223 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5515545606613159 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.853533148765564 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37582629919052124 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_1_of_3‚0.9277312159538269 6validation_average_jet_accuracy‚0.3935440480709076 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6469311118125916 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 *jet/accuracy_1_of_4‚0.9698782563209534 epoch‚31 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32288187742233276 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8027756810188293 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4512482285499573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3173768222332001 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.644597053527832 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.31418779492378235 /particle/accuracy_4_of_4‚0.9671996831893921 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6304981112480164 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7348777055740356 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8399041891098022 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7155969738960266 *jet/accuracy_1_of_2‚0.7685233354568481 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27513033151626587 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8467766046524048 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3804492652416229 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13412067294120789 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9248095154762268 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.018923506140708923 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7149419784545898 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33613523840904236 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8488327264785767 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024282345548272133 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8760842084884644 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 *jet/accuracy_3_of_4‚0.1148163229227066 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4565688669681549 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8745027184486389 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7403072118759155 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 *particle/specificity‚0.913370668888092 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.382981538772583 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6988449096679688 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.34877532720565796 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6652613878250122 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21823067963123322 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7217475771903992 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33736690878868103 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8479824066162109 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.33559688925743103 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.642066478729248 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8085107803344727 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_2_of_3‚1 % _timestamp‚1.7848555596668289e+09 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7263351082801819 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6502040028572083 _step‚747 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5852574706077576 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3355713486671448 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01824122853577137 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3231620788574219 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025286871939897537 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8779537081718445 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12065960466861725 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9466974139213562 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6984425187110901 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6284666061401367 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26164212822914124 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.010037348605692387 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8183493614196777 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8136183619499207 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2580834925174713 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8779537081718445 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6668866872787476 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2772262394428253 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8482013940811157 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.6097331643104553 _runtime‚16323.02497327 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36964526772499084 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6321908831596375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_2_of_4‚0.8398163914680481 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6881241202354431 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4536079466342926 +particle/sensitivity‚0.6344774961471558 trainer/global_step‚8959 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6135223507881165 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24683678150177002 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7555388808250427 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7250706553459167 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05264808610081673 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15172159671783447 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2559436857700348 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6738651990890503 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6527370810508728 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2648586630821228 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10567617416381836 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7687284350395203 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487582325935364 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3231620788574219 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04090919345617294 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23528528213500977 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6910951733589172 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7618670463562012 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6991307139396667 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6647914052009583 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.049635499715805054 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.010037348605692387 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.138003408908844 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21483798325061798 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24528050422668457 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.803298830986023 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7691358923912048 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 +jet/accuracy_2_of_2‚0.16672135889530182 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.642066478729248 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06981723755598068 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 +jet/accuracy_1_of_1‚0.35920223593711853 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5046771168708801 +jet/accuracy_3_of_3‚0.07540739327669144 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2327897846698761 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6386305689811707 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6942816376686096 )loss/val_total_loss‚7.839344682318377 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7739335298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6034317016601562 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2724322974681854 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.13881085813045502 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4949040114879608 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 +È‚;í&þu 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.382981538772583 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6988449096679688 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.34877532720565796 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6652613878250122 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21823067963123322 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7217475771903992 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33736690878868103 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8479824066162109 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.33559688925743103 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.642066478729248 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8085107803344727 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_2_of_3‚1 % _timestamp‚1.7848555596668289e+09 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7263351082801819 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6502040028572083 _step‚747 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5852574706077576 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3355713486671448 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01824122853577137 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3231620788574219 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025286871939897537 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8779537081718445 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12065960466861725 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9466974139213562 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6984425187110901 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6284666061401367 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26164212822914124 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.010037348605692387 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8183493614196777 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8136183619499207 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2580834925174713 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8779537081718445 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6668866872787476 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2772262394428253 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8482013940811157 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.6097331643104553 _runtime‚16323.02497327 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36964526772499084 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6321908831596375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_2_of_4‚0.8398163914680481 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6881241202354431 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4536079466342926 +particle/sensitivity‚0.6344774961471558 trainer/global_step‚8959 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6135223507881165 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24683678150177002 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7555388808250427 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7250706553459167 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05264808610081673 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15172159671783447 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2559436857700348 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6738651990890503 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6527370810508728 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2648586630821228 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10567617416381836 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7687284350395203 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487582325935364 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3231620788574219 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04090919345617294 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23528528213500977 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6910951733589172 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7618670463562012 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6991307139396667 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6647914052009583 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.049635499715805054 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.010037348605692387 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.138003408908844 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21483798325061798 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24528050422668457 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.803298830986023 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7691358923912048 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 +jet/accuracy_2_of_2‚0.16672135889530182 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.642066478729248 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06981723755598068 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 +jet/accuracy_1_of_1‚0.35920223593711853 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5046771168708801 +jet/accuracy_3_of_3‚0.07540739327669144 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2327897846698761 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6386305689811707 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6942816376686096 )loss/val_total_loss‚7.839344682318377 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7739335298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6034317016601562 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2724322974681854 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.13881085813045502 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4949040114879608 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6367247700691223 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5515545606613159 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.853533148765564 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37582629919052124 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_1_of_3‚0.9277312159538269 6validation_average_jet_accuracy‚0.3935440480709076 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6469311118125916 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 *jet/accuracy_1_of_4‚0.9698782563209534 epoch‚31 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32288187742233276 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8027756810188293 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4512482285499573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3173768222332001 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.644597053527832 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.31418779492378235 /particle/accuracy_4_of_4‚0.9671996831893921 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6304981112480164 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7348777055740356 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8399041891098022 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7155969738960266 *jet/accuracy_1_of_2‚0.7685233354568481 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27513033151626587 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8467766046524048 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3804492652416229 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13412067294120789 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46212121844291687 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9248095154762268 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.018923506140708923 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7149419784545898 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33613523840904236 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8488327264785767 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024282345548272133 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8760842084884644 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 *jet/accuracy_3_of_4‚0.1148163229227066 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4565688669681549 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8745027184486389 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7403072118759155 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 *particle/specificity‚0.913370668888092ë±è÷^î&‚(:Ä  “øŠÓ¡—Ö³ disk.drbd1965.in‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2944.66015625 cpu‚25.333984697870868* proc.memory.percent‚11.981852849324543 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1817.o]…ñut‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚47068.51806640625$ disk./.usageGB‚1260.857608795166 disk.drbd1817.in‚0 disk.sda2.out‚ 740.90625 disk.drbd1289.out‚0 disk.drbd1938.out‚0( proc.memory.availableMB‚ 1936.2890625 disk.drbd1584.out‚0 network.recv‚ 1.124387e+06) disk./.usagePercent‚70.51375801228662 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.rbd1.out‚ 84.6494140625 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 network.sent‚ 1.5768295e+07 disk.dm-0.in‚ 101.85546875 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 740.90625 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0# disk.dm-0.out‚47068.51806640625 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625$ memory_percent‚92.12121963500977 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0ï )´žï&‚(:“ “øŠÓö×¼³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚280.166' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.38866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬9Lð&‚(:’ ¢øŠÓÿÇß³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚151.36' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚50.45333333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùm<‹Rñ&‚(:Ç  ¢øŠÓîàü³ disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1285.in‚0 network.sent‚ 1.5822272e+07 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 101.85546875 disk.nvme0n1.in‚ 101.85546875 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 network.recv‚ 1.127182e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 741.2109375 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2944.66015625 disk.drbd1828.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51366243693393 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.sda.out‚ 741.2109375 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0# memory_percent‚92.1213467915853 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚47068.72509765625 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0$ disk./.usageGB‚1260.855899810791 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 cpu‚25.333937344556126 disk.drbd1817.in‚0 disk.rbd1.out‚ 84.7119140625 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1936.2578125 disk.drbd1673.out‚0# disk.dm-0.out‚47068.72509765625* proc.memory.percent‚11.981852849324543 disk.drbd1584.out‚0’ÅGŸò&‚(:” ±øŠÓ”¯ç³ gpu.0.gpu‚61.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.638' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.879333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»—)Ró&‚(:Ç  ±øŠÓ‹”„´) disk./.usagePercent‚70.51366243693393$ disk./.usageGB‚1260.855899810791 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1936.24609375 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 network.sent‚ 1.5825271e+07 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 741.546875 disk.drbd1931.out‚0 disk.drbd1289.out‚0# proc.memory.rssMB‚ 2944.66015625& disk.nvme0n1.out‚47069.00634765625 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.sda.in‚ 0.03515625 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1552.out‚0 network.recv‚ 1.128354e+06 disk.sda.out‚ 741.546875 disk.dm-0.in‚ 101.85546875 disk.drbd1510.in‚0 disk.drbd1552.in‚0 cpu‚25.333860382952757 disk.drbd1931.in‚0 disk.drbd1550.in‚0# disk.dm-0.out‚47069.00634765625 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 84.7119140625 disk.drbd1285.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1828.out‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.981852849324543 disk.drbd2078.in‚0 disk.drbd2041.in‚0$ memory_percent‚92.12139447530112ë0ÍVô&‚(:Ë  ÀøŠÓÃÁú³ disk.drbd2078.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2944.66015625 proc.cpu.threads‚17# disk.dm-0.out‚47069.54150390625 disk.drbd2118.out‚0 disk.sda1.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0$ memory_percent‚92.12139447530112 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚47069.54150390625 disk.drbd1289.in‚0 cpu‚25.333799222737273) disk./.usagePercent‚70.51366585033938 disk.sda2.out‚ 741.89453125 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 network.sent‚ 1.5828274e+07 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 network.recv‚ 1.129486e+06 disk.drbd1938.out‚0% disk./.usageGB‚1260.8559608459473 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 84.923828125 disk.sda.in‚ 0.03515625 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 741.89453125 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1936.24609375 disk.drbd1931.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.981852849324543 disk.drbd1817.out‚0­õSžõ&‚(:“ ÀøŠÓÖ£Û³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚291.862' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.28733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡&{ú4ö&/ _wandbruntime‚16390  _runtime‚16390˜mkÇ÷&Á trainer/global_step‚8999 $ _timestamp‚1.784855627639028e+09 _runtime‚16390.996247114 _step‚748 & lr-AdamW/pg1‚3.438129404198799e-05 & lr-AdamW/pg2‚3.438129404198799e-05šfPbÌø&Æ & lr-AdamW/pg1‚3.438129404198799e-05 & lr-AdamW/pg2‚3.438129404198799e-05 trainer/global_step‚8999 $ _timestamp‚1.784855627639028e+09 _runtime‚16390.996247114 _step‚748ìNÆÏî4ù&/ _wandbruntime‚16391  _runtime‚16391ÜŠ"Å(ú&‚P JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161761239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83581937 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23985252 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚720527523791872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2592213353793536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚326 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51416984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚720527945247232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83581937 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚709036841204736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11491104042496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2601606681521152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚8999 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83592001 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27432063 # _timestamp‚1.78485562764007e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9378969922818 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78169238 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2601606681521152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2592213327620096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27431737 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2591190768543472 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51417315 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2600569763894258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9393353901056Ž”"° (û&‡P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51417315 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2600569763894258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9393353901056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161761239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83581937 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23985252 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚720527523791872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2592213353793536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚326 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51416984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚720527945247232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83581937 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚709036841204736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11491104042496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2601606681521152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚8999 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83592001 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27432063 # _timestamp‚1.78485562764007e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9378969922818 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78169238 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2601606681521152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2592213327620096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27431737 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2591190768543472 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181íJ$ÿX4ü&/ _wandbruntime‚16392  _runtime‚16392óEŸÞéý&N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51422586 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_ooåH´*m_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2600849586469786 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2592492263084032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27435023 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23987887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9394376409088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9378646113172 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2601887984936448 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2592492236910592 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2601886613319680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161768618 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83601097 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161778702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9379990724570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83591019 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2591469570317248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83591019 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9395721852416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _step‚750 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161778702 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2601887984936448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚709115427570688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚8999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11492312171520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784855629111685e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78177599 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161768618 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78177605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚720607784710656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2592492263084032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2592492236910592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚329 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050.0'þ&’N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚709115059479552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11492357139968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51422586 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2600849586469786 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2592492263084032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27435023 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23987887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9394376409088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9378646113172 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2601887984936448 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2592492236910592 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2601886613319680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161768618 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83601097 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161778702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9379990724570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83591019 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2591469570317248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83591019 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9395721852416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _step‚750 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161778702 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2601887984936448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚709115427570688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚8999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11492312171520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784855629111685e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78177599 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161768618 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78177605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚720607784710656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2592492263084032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2592492236910592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚329 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 EfmM‰'‚(:  üøŠÓî±Ü³ disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2944.66015625( proc.memory.availableMB‚ 1936.2265625 network.sent‚ 1.5898066e+07 disk.sda.in‚ 0.03515625 disk.sda.out‚ 743.34375 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚47071.5107421875 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0# memory_percent‚92.1214739481608* proc.memory.percent‚11.981852849324543 proc.cpu.threads‚17" disk.dm-0.out‚47071.5107421875 disk.rbd1.out‚ 85.1123046875 disk.drbd1931.in‚0 disk.drbd1226.out‚0 network.recv‚ 1.135385e+06 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1584.in‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51377507931389 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 743.34375 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1510.in‚0 cpu‚25.333555468507917% disk./.usageGB‚1260.8579139709473 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0YΓ‚UŠ'‚(:Ê  ‹ùŠÓ×Þ…´ disk.drbd2078.out‚0# proc.memory.rssMB‚ 2944.66015625 cpu‚25.333523902723652 disk.drbd2118.out‚0 disk.drbd1510.in‚0 network.sent‚ 1.5900576e+07 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 85.2060546875) disk./.usagePercent‚70.51377507931389 disk.drbd1226.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚47072.2607421875 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.981852849324543 disk.nvme0n1.in‚ 101.85546875 disk.drbd2118.in‚0 disk.sda.out‚ 743.73828125 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0) proc.memory.availableMB‚ 1935.97265625 disk.drbd1817.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.12250709533691% disk.nvme0n1.out‚47072.2607421875 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 743.73828125 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 network.recv‚ 1.136471e+06% disk./.usageGB‚1260.8579139709473 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0’PR‘ž‹'‚(:“ ‹ùŠÓ‘˜è³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.382' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.12733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ¡,*“Œ'‚(:ˆ šùŠÓ¼Ô³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.868' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸчT'‚(:É  šùŠÓ´Öî³ disk.drbd1584.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.sda.out‚ 744.0703125 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0 cpu‚25.33344857241626 disk.rbd1.out‚ 85.2685546875 disk.drbd1550.in‚0# disk.dm-0.out‚47072.77099609375 proc.cpu.threads‚17% disk./.usageGB‚1260.8579139709473 disk.drbd1226.out‚0 disk.sda2.in‚ 0.03515625* proc.memory.percent‚11.981852849324543 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 744.0703125 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2944.66015625 disk.drbd1673.out‚0 network.recv‚ 1.137551e+06 disk.drbd1828.in‚0 disk.drbd1673.in‚0$ memory_percent‚92.12250709533691 disk.drbd1285.in‚0& disk.nvme0n1.out‚47072.77099609375 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 101.85546875 disk.drbd1356.out‚0 disk.drbd1938.in‚0) disk./.usagePercent‚70.51377507931389 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1935.97265625 network.sent‚ 1.5903573e+07 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0²u€!4Ž'/ _wandbruntime‚16477  _runtime‚16477Ëþ œÈ' _runtime‚16477.370838285 _step‚752 & lr-AdamW/pg1‚3.377222466180613e-05 & lr-AdamW/pg2‚3.377222466180613e-05 trainer/global_step‚9049 % _timestamp‚1.7848557140127785e+09ÛÍC™Í'Ç & lr-AdamW/pg1‚3.377222466180613e-05 & lr-AdamW/pg2‚3.377222466180613e-05 trainer/global_step‚9049 % _timestamp‚1.7848557140127785e+09 _runtime‚16477.370838285 _step‚752ð®'È4‘'/ _wandbruntime‚16477  _runtime‚16477 ªÞK’'ƒP UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9445888893440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2606158818318336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9445888893440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78587582 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 trainer/global_step‚9049 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78587588 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2615603335602176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2606158818318336`ùľ YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162625471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27579942 $ _timestamp‚1.784855714013589e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2606158792144896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84037889 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27579592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11553598140928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚712965791411200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84047953 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2615603335602176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚753 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84047953 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162635541 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚724519389552128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2605130882660240 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11553659893760 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2606158792144896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84037889 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27579592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11553598140928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚712965791411200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84047953 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2615603335602176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚753 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84047953 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162635541 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚724519389552128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2605130882660240 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11553659893760 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _runtime‚16478.845311746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51702749 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2614839409002028 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11554894209536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2615884639017472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2606437701435392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27582857 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27582518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78595949 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚713044745868288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51702405 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2606437727608832 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2615883267400704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848557154876654e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78595955 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84057049 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162653004 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162642920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9429749995980 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2615884639017472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ƒ±æÉ–'“N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9446911408640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24119887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚724599218629632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162642920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9429749995980 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2615884639017472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚713044377777152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2606437701435392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _runtime‚16478.845311746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51702749 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2614839409002028 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11554894209536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2615884639017472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2606437701435392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27582857 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27582518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78595949 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚713044745868288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51702405 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.currentc­³P‚53357056 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2606437727608832 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2615883267400704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848557154876654e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78595955 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84057049 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162653004 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84493841 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚716895109708800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2629601361292800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51975356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2619070945921072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84503905 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2629599989683200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9481177688434 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9479833077148 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9495733013504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2620104282843136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27725749 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163499773 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79005938 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2628552149037474 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79005932 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2629599989683200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2620104256669696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2619070971349040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚9099 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84503905 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2629601361292800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9495733013504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9497078449664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _runtime‚16564.354939372 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163509843 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728511103079936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51974996 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11616044638208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2628550778998220 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24249247 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24249252 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚757\15" «'ˆP ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2619070945921072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84503905 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2629599989683200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9481177688434 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9479833077148 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9495733013504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2620104282843136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27725749 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163499773 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79005938 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.currenl5|ìt‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2628552149037474 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79005932 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2629599989683200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2620104256669696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2619070971349040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚9099 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84503905 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2629601361292800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9495733013504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9497078449664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _runtime‚16564.354939372 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163509843 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728511103079936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51974996 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11616044638208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2628550778998220 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24249247 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24249252 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚757 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84493841 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚716895109708800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2629601361292800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51975356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0õ",Û4¬'/ _wandbruntime‚16565  _runtime‚16565ŽÿóD'­'N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9480853878788 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716974064165888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _step‚758 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163527306 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11617243777536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163517222 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2620383165960192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2628831971613002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79014305 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2620383192133632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51980924 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84502923 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2629879921481728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728591363397632 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84513001 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9498100964864 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2619349747694848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2620383192133632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9496755521536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84502923 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.784855802471914e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51980570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚16565.829578782 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163527306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27729032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27728683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚9099 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2619349773122816 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2620383165960192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79014305 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11617299231744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0k<·}'®'’N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _step‚758 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163527306 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11617243777536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163517222 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2620383165960192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2628831971613002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79014305 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2620383192133632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51980924 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84502923 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2629879921481728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728591363397632 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84513001 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9498100964864 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2619349747694848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2620383192133632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9496755521536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84502923 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.784855802471914e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51980570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚16565.829578782 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163527306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27729032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27728683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚9099 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2619349773122816 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2620383165960192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79014305 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11617299231744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9480853878788 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716974064165888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354öä0^4¯'/ _wandbruntime‚16565  _runtime‚16565‚¶j °'„ 0loss/FRt2/assignment_loss‚0.6482822547125262 2loss/SRqqt1/assignment_loss‚0.4617138829239911 3loss/FBt1/assignment_loss‚0.0007904224065242715 /loss/FRt1/detection_loss‚0.5765493157801307 /loss/FRt2/detection_loss‚1.5293871000885617 0loss/SRqqt2/detection_loss‚0.636589876333137 2loss/FBt2/assignment_loss‚0.032324544688856496 1loss/SRqqt1/detection_loss‚0.8268894199076701 %loss/total_loss‚7.818624413768428 _step‚759 2loss/SRqqt2/assignment_loss‚0.4777445499471639 epoch‚32 trainer/global_step‚9099 % _timestamp‚1.7848558025064979e+09 0loss/FRt1/assignment_loss‚1.5400922929999805 .loss/FBt1/detection_loss‚0.632988313363204 /loss/FBt2/detection_loss‚0.4552724406166817 _runtime‚16565.863742404„Ú±'‰ 2loss/SRqqt2/assignment_loss‚0.4777445499471639 epoch‚32 trainer/global_step‚9099 % _timestamp‚1.7848558025064979e+09 0loss/FRt1/assignment_loss‚1.5400922929999805 .loss/FBt1/detection_loss‚0.632988313363204 /loss/FBt2/detection_loss‚0.4552724406166817 _runtime‚16565.863742404 0loss/FRt2/assignment_loss‚0.6482822547125262 2loss/SRqqt1/assignment_loss‚0.4617138829239911 3loss/FBt1/assignment_loss‚0.0007904224065242715 /loss/FRt1/detection_loss‚0.5765493157801307 /loss/FRt2/detection_loss‚1.5293871000885617 0loss/SRqqt2/detection_loss‚0.636589876333137 2loss/FBt2/assignment_loss‚0.032324544688856496 1loss/SRqqt1/detection_loss‚0.8268894199076701 %loss/total_loss‚7.818624413768428 _step‚759÷åW>õS²'‚(:È  ƒúŠÓ€Þ‚´ network.recv‚ 1.146475e+06 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1936.80859375 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1552.out‚0& disk.nvme0n1.out‚47075.65380859375 disk.sda.in‚ 0.03515625 cpu‚25.333101144050254) disk./.usagePercent‚70.51386745459898 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 85.6689453125 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 network.sent‚ 1.5981856e+07% disk./.usageGB‚1260.8595657348633# proc.memory.rssMB‚ 2944.66015625$ memory_percent‚92.11910565694174 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 746.671875# disk.dm-0.out‚47075.65380859375 disk.nvme0n1.in‚ 101.85546875 disk.drbd2118.in‚0 disk.sda.out‚ 746.671875 disk.drbd2041.in‚0* proc.memory.percent‚11.981852849324543 proc.cpu.threads‚17&—µœ³'‚(:‘ ƒúŠÓþƒã³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚107.798' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.93266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚083UÜ“´'‚(:ˆ ’úŠÓ…êγ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚287.937' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.979 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Û$ù™µ'‚(:Å  ’úŠÓœ«ò³ disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1356.out‚0 network.sent‚ 1.6041913e+07 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 cpu‚25.33302655486067 disk.drbd1285.in‚0* proc.memory.percent‚11.981852849324543 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2944.66015625% disk./.usageGB‚1260.8595657348633 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625% proc.memory.availableMB‚ 1936.6875) disk./.usagePercent‚70.51386745459898 disk.drbd1510.in‚0 disk.´5·drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1584.in‚0" disk.dm-0.out‚47075.9169921875 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 85.7314453125 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 747.03515625 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 101.85546875% disk.nvme0n1.out‚47075.9169921875 disk.sda.out‚ 747.03515625 disk.rbd1.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.11959838867188 network.recv‚ 1.148998e+06 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0Õ7üfŸ¶'‚(:” ¡úŠÓ¡Ç©³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.888' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.96266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥~úŠS·'‚(:È  ¡úŠÓÝȳ disk.rbd1.out‚ 85.7314453125 disk.nvme0n1.in‚ 101.85546875 disk.drbd1817.out‚0 network.recv‚ 1.149967e+06 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0' proc.memory.availableMB‚ 1936.546875 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2944.66015625) disk./.usagePercent‚70.51386745459898 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 network.sent‚ 1.6044957e+07 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0$ memory_percent‚92.12017059326172 proc.cpu.threads‚17" disk.dm-0.out‚47076.1474609375 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.sda.out‚ 747.49609375 cpu‚25.332967786497637% disk./.usageGB‚1260.8595657348633 disk.sda2.out‚ 747.49609375 disk.drbd2078.out‚0 disk.drbd2041.out‚0% disk.nvme0n1.out‚47076.1474609375 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 101.85546875 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.981852849324543 disk.drbd1931.out‚0î1ùM¸'‚(:  °úŠÓúúó³# disk.nvme0n1.out‚47076.55859375 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.dm-0.out‚47076.55859375 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.sda2.in‚ 0.03515625 disk.sda1.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1936.4921875 network.recv‚ 1.1509e+06) disk./.usagePercent‚70.51386766793681 disk.drbd1552.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2944.66015625 disk.drbd1965.in‚0 disk.sda2.out‚ 747.80078125 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.981852849324543 disk.sda.in‚ 0.03515625% disk./.usageGB‚1260.8595695495605 disk.nvme0n1.in‚ 101.85546875 disk.drbd1226.out‚0 disk.dm-0.in‚ 101.85546875 network.sent‚ 1.6048494e+07 disk.sda.out‚ 747.80078125 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 85.763671875 disk.drbd1285.out‚0 cpu‚25.332921984872478 disk.drbd1931.out‚0$ memory_percent‚92.12039311726889ê tEž¹'‚(:“ °úŠÓ°î̳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.185' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.06166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñüUŸº'‚(:” ¿úŠÓþ¹â³ gpu.0.gpu‚63.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.397' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.46566666666665 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ö›M»'‚(:  ¿úŠÓ˜Ëÿ³ network.sent‚ 1.6051003e+07 disk.drbd1828.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0) proc.memory.availableMB‚ 1936.43359375 proc.cpu.threads‚17 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 748.2890625 disk.sda2.out‚ 748.2890625* proc.memory.percent‚11.981852849324543 network.recv‚ 1.151977e+06 disk.drbd2041.in‚0 disk.drbd2111.in‚0# disk.nvme0n1.out‚47076.74609375 disk.sda2.in‚ 0.03515625 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.dm-0.out‚47076.74609375 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8595695495605 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 85.857421875 disk.drbd1285.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2944.66015625) disk./.usagePercent‚70.51386766793681 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0$ memory_percent‚92.12063153584799 cpu‚25.33286207445814 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1552.out‚0±JŒíŸ¼'‚(:” ÎúŠÓ“‹Ì³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚142.895' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚47.631666666666675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k®’®U½'‚(:Ê  ÎúŠÓŨè³ disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0) proc.memory.availableMB‚ 1936.68359375 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.sda.out‚ 748.56640625 disk.drbd1673.in‚0& disk.nvme0n1.out‚47077.11376953125% disk./.usageGB‚1260.8595695495605 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 85.919921875 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 network.sent‚ 1.6054002e+07 network.recv‚ 1.153003e+06 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0# disk.dm-0.out‚47077.11376953125 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0# memory_percent‚92.1196142832438 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 748.56640625 disk.nvme0n1.in‚ 101.85546875 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1510.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 101.85546875* proc.memory.percent‚11.981852849324543) disk./.usagePercent‚70.51386766793681 disk.drbd2111.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2944.66015625 cpu‚25.332773167642753ÁÈk4¾'/ _wandbruntime‚16650  _runtime‚16650Þ÷ŸÊ¿'Ä _runtime‚16650.202285166 _step‚760 ' lr-AdamW/pg1‚3.2563914412885214e-05 ' lr-AdamW/pg2‚3.2563914412885214e-05 trainer/global_step‚9149 % _timestamp‚1.7848558868450532e+09mä°+ÏÀ'É ' lr-AdamW/pg1‚3.2563914412885214e-05 ' lr-AdamW/pg2‚3.2563914412885214e-05 trainer/global_step‚9149 % _timestamp‚1.7848558868450532e+09 _runtime‚16650.202285166 _step‚760øÇžÎ4Á'/ _wandbruntime‚16650  _runtime‚16650Rwî (Â'ƒP WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24381247 % _timestamp‚1.7848558868461046e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚732503243959808 trainer/global_step‚9149 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24381252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2633011060037840 _runtime‚16650.20694188 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720824428006400 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2633011034609872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11678390303744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9548268005888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164384145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2642543341609082 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84959857 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9548268005888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2634049747367936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2643598015373824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11678447862272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84959857 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27873343 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9532281571242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9530936959956 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79424288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9546922569728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 _step‚761 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2643596643764224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2643596643764224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84949793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52254222 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27872975Àv±š(Ã'ˆP ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2634049721194496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2643598015373824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚732502818310144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚732503243959808 trainer/global_step‚9149 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24381252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2633011060037840 _runtime‚16650.20694188 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720824428006400 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2633011034609872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11678390303744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9548268005888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164384145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2642543341609082 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84959857 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9548268005888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2634049747367936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2643598015373824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11678447862272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84959857 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27873343 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9532281571242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9530936959956 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79424288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9546922569728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 _step‚761 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚373 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2643596643764224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2643596643764224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84949793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52254222 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27872975 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24381247 % _timestamp‚1.7848558868461046e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9549290521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2633289836383648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9549290521088 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMon1Jitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9547945077760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720903382463488 _runtime‚16651.673650185 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11679631395840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2642823164184610 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9531957761596 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79432649 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2643876575562752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784855888316079e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79432655 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9533302372994 trainer/global_step‚9149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2643877947179520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2642821794145244 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2643877947179520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11679686850048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164401608 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52260143 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164391524 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚732582645768192 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27876251 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164401608 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84968953 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2634328656658432 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2634328656658432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79432649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24383892 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24383887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84968953 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84958875 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27875886 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚732583069313536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _step‚762 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52259773 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720903014372352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2633289861811616°üö°'Æ'’N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24383887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84968953 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84958875 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27875886 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚732583069313536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _step‚762 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52259773 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720903014372352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2633289861811616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9549290521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2633289836383648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9549290521088 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9547945077760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720903382463488 _runtime‚16651.673650185 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11679631395840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2642823164184610 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9531957761596 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79432649 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2643876575562752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784855888316079e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79432655 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9533302372994 trainer/global_step‚9149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2643877947179520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2642821794145244 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2643877947179520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11679686850048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164401608 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52260143 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164391524 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚732582645768192 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27876251 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164401608 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84968953 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2634328656658432 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2634328656658432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79432649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24383892 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655úf¶4Ç'/ _wandbruntime‚16651  _runtime‚16651%-ê È'‡ /loss/FBt1/detection_loss‚0.8040529660199984 %loss/total_loss‚7.984184618225946 2loss/SRqqt2/assignment_loss‚0.5297042487485311 1loss/SRqqt1/detection_loss‚0.8274178675973949 1loss/SRqqt2/detection_loss‚0.5957048864691313 /loss/FBt2/detection_loss‚0.3971195009517326 % _timestamp‚1.7848558883557026e+09 _runtime‚16651.712916651 _step‚763 0loss/FRt2/assignment_loss‚0.6832988167543547 2loss/SRqqt1/assignment_loss‚0.5276225738547359 4loss/FBt1/assignment_loss‚2.3147732286442026e-05 /loss/FRt1/detection_loss‚0.5324262697463928 /loss/FRt2/detection_loss‚1.4884385511514144 epoch‚32 trainer/global_step‚9149 0loss/FRt1/assignment_loss‚1.5917666441521672 2loss/FBt2/assignment_loss‚0.006609145047805853#LíÉ'Œ 1loss/SRqqt2/detection_loss‚0.5957048864691313 /loss/FBt2/detection_loss‚0.3971195009517326 % _timestamp‚1.7848558883557026e+09 _runtime‚16651.712916651 _step‚763 0loss/FRt2/assignment_loss‚0.6832988167543547 2loss/SRqqt1/assignment_loss‚0.5276225738547359 4loss/FBt1/assignment_loss‚2.3147732286442026e-05 /loss/FRt1/detection_loss‚0.5324262697463928 /loss/FRt2/detection_loss‚1.4884385511514144 epoch‚32 trainer/global_step‚9149 0loss/FRt1/assignment_loss‚1.5917666441521672 2loss/FBt2/assignment_loss‚0.006609145047805853 /loss/FBt1/detection_loss‚0.8040529660199984 %loss/total_loss‚7.984184618225946 2loss/SRqqt2/assignment_loss‚0.5297042487485311 1loss/SRqqt1/detection_loss‚0.8274178675973949ûAÈ¢ žÊ'‚(:“ ÝúŠÓ‹å´³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.336' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.11200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Å©ýTË'‚(:É  ÝúŠÓìøѳ disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 748.89453125 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 network.recv‚ 1.15398e+06 disk.drbd1938.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.981852849324543) disk./.usagePercent‚70.51386766793681 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1931.out‚0 disk.drbd1938.out‚0& disk.nvme0n1.out‚47077.30517578125 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0# memory_percent‚92.1207586924235 network.sent‚ 1.6057001e+07) proc.memory.availableMB‚ 1936.40234375 cpu‚25.332759254996045 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.sda.out‚ 748.89453125 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2944.66015625 proc.cpu.threads‚17% disk./.usageGB‚1260.8595695495605 disk.drbd1965.out‚0 disk.rbd1.out‚ 85.919921875 disk.drbd1817.out‚0# disk.dm-0.out‚47077.30517578125 disk.drbd1828.in‚0 disk.drbd1552.out‚0ÆŽBŸÌ'‚(:” ìúŠÓø†¿³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.335' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.11166666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zôwÕRÍ'‚(:Ç  ìúŠÓ¼Äã³ disk.sda2.out‚ 749.109375$ memory_percent‚92.11974143981934 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚47077.66357421875 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 749.109375 disk.drbd2078.in‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2944.66015625 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.sda2.in‚ 0.03515625# disk.dm-0.out‚47077.66357421875 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 85.9521484375 disk.drbd1550.out‚0 disk.sda1.out‚0% disk./.usageGB‚1260.8595695495605 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 network.sent‚ 1.6117701e+07 disk.drbd1931.in‚0 disk.drbd1584.out‚0) proc.memory.availableMB‚ 1936.65234375 network.recv‚ 1.156604e+06 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.981852849324543 disk.drbd1356.out‚0 disk.drbd1552.in‚0 cpu‚25.33274437770013) disk./.usagePercent‚70.51386766793681 disk.drbd1817.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17µG*{TÎ'‚(:É  ûúŠÓÁºü³ disk.rbd1.out‚ 86.2255859375 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0& disk.nvme0n1.out‚47077.84716796875 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2944.66015625 cpu‚25.332714395192543 disk.dm-0.in‚ 101.85546875 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1931.in‚0# memory_percent‚92.1207586924235 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.981852849324543 disk.drbd1938.out‚0# disk.dm-0.out‚47077.84716796875 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 749.5859375% disk./.usageGB‚1260.8595695495605 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1936.40234375 network.sent‚ 1.6120165e+07 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 749.5859375 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 network.recv‚ 1.157545e+06 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.sda2.in‚ 0.03515625) disk./.usagePercent‚70.51386766793681 disk.drbd1965.out‚0 disk.drbd2078.out‚0v¿ü,žÏ'‚(:“ ûúŠÓ´¶Ý³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.618' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.20599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!N,žÐ'‚(:“ ŠûŠÓ»½¾³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚288.752' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.25066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\yVÑ'‚(:Ë  ŠûŠÓª•Û³ disk.drbd1673.in‚0 network.recv‚ 1.158586e+06 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0) disk./.usagePercent‚70.51376867917867 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 749.87109375 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.rbd1.out‚ 86.2880859375 disk.drbd1817.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚47078.07373046875 disk.sda1.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 cpu‚25.332656049608808 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1550.in‚0% disk./.usageGB‚1260.8577995300293 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1936.61328125 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 749.87109375 network.sent‚ 1.612317e+07# disk.dm-0.out‚47078.07373046875 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0$ memory_percent‚92.11990038553873# proc.memory.rssMB‚ 2944.66015625* proc.memory.percent‚11.981852849324543 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.sda1.out‚0qfÙfŸÒ'‚(:” ™ûŠÓÌøг gpu.0.gpu‚53.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.728' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.24266666666668 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³cwSÓ'‚(:È  ™ûŠÓ¦ˆñ³ disk.drbd1938.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1828.out‚0* proc.memory.percent‚11.981852849324543 disk.drbd1828.in‚0 cpu‚25.33261138362289 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚47078.22607421875 network.sent‚ 1.6126217e+07 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.out‚0 disk.rbd1.out‚ 86.2880859375 disk.sda.out‚ 750.4296875 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51376867917867% disk./.usageGB‚1260.8577995300293 disk.drbd1584.in‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚47078.22607421875 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 101.85546875# proc.memory.rssMB‚ 2944.66015625 disk.drbd1965.out‚0 network.recv‚ 1.159663e+06 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0( proc.memory.availableMB‚ 1936.5390625 disk.drbd1584.out‚0 disk.sda2.out‚ 750.4296875 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0$ memory_percent‚92.12020238240561 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17wsYo4Ô'/ _wandbruntime‚16736  _runtime‚16736×UƒGÈÕ' ' lr-AdamW/pg1‚3.1964831925936005e-05 ' lr-AdamW/pg2‚3.1964831925936005e-05 trainer/global_step‚9199 $ _timestamp‚1.784855973392286e+09 _runtime‚16736.74953631 _step‚764ú`ÃÍÖ'Ç trainer/global_step‚9199 $ _timestamp‚1.784855973392286e+09 _runtime‚16736.74953631 _step‚764 ' lr-AdamW/pg1‚3.1964831925936005e-05 ' lr-AdamW/pg2‚3.1964831925936005e-05üƒakê4×'/ _wandbruntime‚16736  _runtime‚167367KR Ø'ƒP VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52534666 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85415809 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚724753746304000 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2646951148726640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reser7ÆLîved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2657594669454848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24513252 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9599457562112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28021021 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11740697155584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2656533164141436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9583385454050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2657593297845248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚736494385901056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9598112125952 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2657594669454848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24513252 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9599457562112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28021021 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11740697155584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2656533164141436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9583385454050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2657593297845248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚736494385901056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9598112125952 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2648274095009792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚387 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚382 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9599134633984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2648274121183232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚9199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 % _timestamp‚1.7848559748684435e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2648274095009792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16738.226048838 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2657874601260544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79851005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79851005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9583061644404 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2648274121183232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85414827 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85414827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52539819 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79850999 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2657873229643776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2647229925072448 _step‚766 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165265826 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2656812986716852 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24515892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52540206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2657874601260544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11741934884864 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2647229950500416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85424905 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165265826 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165275910 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85424905 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28023932 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚736574635645952´’Çæ…Ü'“N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79851005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79851005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9583061644404 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2648274121183232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85414827 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85414827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52539819 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79850999 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2657873229643776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2647229925072448 _step‚766 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165265826 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2656812986716852 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24515892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52540206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2657874601260544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11741934884864 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2647229950500416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85424905 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165265826 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165275910 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85424905 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28023932 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚736574635645952 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11741873139200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9584406255802 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2648274095009792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚387 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚382 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9599134633984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2648274121183232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 trainer/global_step‚9199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 % _timestamp‚1.7848559748684435e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2648274095009792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16738.226048838 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2657874601260544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824þŽ"Rv4Ý'/ _wandbruntime‚16738  _runtime‚16738«‚mêÞ' 2loss/SRqqt2/assignment_loss‚0.5317653619069392 trainer/global_step‚9199 $ _timestamp‚1.784855974905083e+09 1loss/SRqqt1/assignment_loss‚0.609614121602795 3loss/FBt2/assignment_loss‚0.0071759749135397585 .loss/FRt2/detection_loss‚1.400906077076664 1loss/SRqqt1/detection_loss‚0.8942973862216381 _runtime‚16738.262311501 1loss/FBt1/assignment_loss‚0.01376074653269232 /loss/FRt1/detection_loss‚0.5321108941733439 /loss/FBt2/detection_loss‚0.4058420610304701 /loss/FRt1/assignment_loss‚1.544707958731061 0loss/FRt2/assignment_loss‚0.6348082008985204 1loss/SRqqt2/detection_loss‚0.6666505510801531 /loss/FBt1/detection_loss‚0.7773100730233096 %loss/total_loss‚8.018949407191126 epoch‚32 _step‚767“8¿ ß'† 1loss/SRqqt1/assignment_loss‚0.609614121602795 3loss/FBt2/assignment_loss‚0.0071759749135397585 .loss/FRt2/detection_loss‚1.400906077076664 1loss/SRqqt1/detection_loss‚0.8942973862216381 _runtime‚16738.262311501 1loss/FBt1/assignment_loss‚0.01376074653269232 /loss/FRt1/detection_loss‚0.5321108941733439 /loss/FBt2/detection_loss‚0.4058420610304701 /loss/FRt1/assignment_loss‚1.544707958731061 0loss/FRt2/assignment_loss‚0.6348082008985204 1loss/SRqqt2/detection_loss‚0.6666505510801531 /loss/FBt1/detection_loss‚0.7773100730233096 %loss/total_loss‚8.018949407191126 epoch‚32 _step‚767 2loss/SRqqt2/assignment_loss‚0.5317653619069392 trainer/global_step‚9199 $ _timestamp‚1.784855974905083e+09ÿÒi+Ä“à'‚(:ˆ ¨ûŠÓò«³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚274.188' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cc›´Lá'‚(:Á  ¨ûŠÓ԰ʳ) disk./.usagePercent‚70.51388174823431 disk.dm-0.in‚ 101.85546875 disk.drbd2041.out‚0 network.recv‚ 1.16058e+06 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0$ disk.nvme0n1.out‚47078.958984375 disk.drbd1550.in‚0 cpu‚25.33252353965206 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.12028185526529* proc.memory.percent‚11.981852849324543 disk.sda2.in‚ 0.03515625 disk.drbd1226.in‚0 network.sent‚ 1.6129229e+07 disk.rbd1.out‚ 86.3203125 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17# disk./.usageGB‚1260.85982131958 disk.sda2.out‚ 750.71484375 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2944.66015625 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1936.51953125 disk.drbd1817.in‚0 disk.drbd1289.out‚0! disk.dm-0.out‚47078.958984375 disk.drbd1356.out‚0 disk.sda.out‚ 750.71484375 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1817.out‚0ŽI<“â'‚(:ˆ ·ûŠÓ¡ýÖ³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚283.599' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.533 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù/Nã'‚(:à  ·ûŠÓ¹þø³ disk.drbd1550.in‚0 disk.drbd1289.out‚0( proc.memory.availableMB‚ 1936.4765625 disk.dm-0.in‚ 101.85546875 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 network.sent‚ 1.6189824e+07% disk./.usageGB‚1260.8597564697266 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 network.recv‚ 1.16335e+06 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2944.66015625 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 751.03515625 cpu‚25.332478442475562 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.rbd1.out‚ 86.4140625$ disk.nvme0n1.out‚47079.505859375 disk.sda.out‚ 751.03515625 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.981852849324543 disk.sda2.in‚ 0.03515625$ memory_percent‚92.12045669555664 proc.cpu.threads‚17) disk./.usagePercent‚70.51387812149102 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1552.out‚0! disk.dm-0.out‚47079.505859375 disk.drbd1817.out‚0qãAä'‚(:’ ÆûŠÓ£å¼³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚301.007' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.33566666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó4zøOå'‚(:Ä  ÆûŠÓìÖÞ³ disk.drbd2118.out‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.981852849324543 disk.drbd1226.in‚0 network.recv‚ 1.164421e+06 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚47080.0029296875 disk.sda2.out‚ 751.3515625 disk.drbd1673.out‚0 cpu‚25.332435218070273 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.85546875) disk./.usagePercent‚70.51387812149102% disk./.usageGB‚1260.8597564697266 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 86.4765625 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2944.66015625$ memory_percent‚92.12045669555664 disk.drbd2078.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.sda.out‚ 751.3515625 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0( proc.memory.availableMB‚ 1936.4765625 network.sent‚ 1.6192322e+07" disk.dm-0.out‚47080.0029296875 disk.drbd1510.out‚0 disk.dm-0.in‚ 101.85546875 disk.sda1.out‚0Ô¿ ižæ'‚(:“ ÕûŠÓÿº°³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚263.741' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.91366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:kBOç'‚(:Ä  ÕûŠÓÄäг" disk.dm-0.out‚47080.1826171875 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1584.out‚0 network.sent‚ 1.6195314e+07 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.981852849324543 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.sda1.out‚0 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 751.6484375 disk.drbd1673.in‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0) disk./.usagePercent‚70.51387812149102 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0( proc.memory.availableMB‚ 1936.4765625 disk.drbd2041.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 network.recv‚ 1.165508e+06 disk.rbd1.in‚0% disk.nvme0n1.out‚47080.1826171875% disk./.usageGB‚1260.8597564697266 disk.sda2.out‚ 751.6484375 disk.drbd1828.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.12045669555664# proc.memory.rssMB‚ 2944.66015625 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 86.4765625 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 cpu‚25.332362042666087;2¶Åžè'‚(:“ äûŠÓ†¤â³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.225' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.74166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£VRé'‚(:Ç  äûŠÓÀ‘„´ disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 751.9296875 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 cpu‚25.332302232883315 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0% disk.nvme0n1.out‚47080.7900390625 disk.dm-0.in‚ 101.85546875$ memory_percent‚92.12045669555664 disk.rbd1.out‚ 86.5087890625 disk.drbd1584.in‚0 disk.drbd2118.in‚0( proc.memory.availableMB‚ 1936.4765625 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 101.85546875 proc.cpu.threads‚17* proc.memory.percent‚11.981852849324543 network.recv‚ 1.166554e+06) disk./.usagePercent‚70.51387812149102% disk./.usageGB‚1260.8597564697266 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0" disk.dm-0.out‚47080.7900390625 network.sent‚ 1.6198859e+07 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.sda.out‚ 751.9296875 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2944.66015625 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625¼TwŸê'‚(:” óûŠÓÛ©Ô³ gpu.0.gpu‚50.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚303.611' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.20366666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øûÈSë'‚(:È  óûŠÓ ô³ disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2111.in‚0 disk.drbd1285.in‚0) disk./.usagePercent‚70.51387812149102" disk.dm-0.out‚47081.0166015625 disk.drbd2078.out‚0 network.sent‚ 1.6201364e+07 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17$ memory_percent‚92.11945533752441 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 752.2890625 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 network.recv‚ 1.167555e+06 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚47081.0166015625 disk.drbd1226.in‚0 disk.sda.out‚ 752.2890625 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2944.66015625 cpu‚25.333268012482705* proc.memory.percent‚11.981852849324543 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 86.6025390625 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 101.85546875) proc.memory.availableMB‚ 1936.72265625% disk./.usageGB‚1260.8597564697266 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0›ˆµ-4ì'/ _wandbruntime‚16827  _runtime‚16827›Íø5í'°u >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2510862648487091 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9238861203193665 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6239804625511169 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21539664268493652 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6923638582229614 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7142553925514221 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.360604465007782 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24032875895500183 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6493884921073914 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 *jet/accuracy_1_of_2‚0.7711294293403625 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚7.797055892397236 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.624163031578064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12506315112113953 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22889353334903717 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.654193103313446 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6441582441329956 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 epoch‚32 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699317336082458 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.635172426700592 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6088438630104065 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7275477051734924 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8406385183334351 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2703915238380432 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6756872534751892 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9651569724082947 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6777549982070923 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8564395308494568 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11173006147146225 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25109294056892395 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32572105526924133 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8438156843185425 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8471320867538452 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6985534429550171 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.645849347114563 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5885108709335327 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24599085748195648 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7357410788536072 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021752577275037766 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8127322793006897 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3856492042541504 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2441018968820572 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7276562452316284 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01516555342823267 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8758172392845154 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 *jet/accuracy_2_of_3‚0.6007881760597229 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6493884921073914 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8867693543434143 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7998046278953552 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4489903450012207 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01516555342823267 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2579079270362854 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658285915851593 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022750183939933777 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2587502598762512 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8190836906433105 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7487757802009583 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9290577173233032 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 +jet/accuracy_1_of_1‚0.35301175713539124 % _timestamp‚1.7848560636595554e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7577517628669739 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9467796683311462 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 _runtime‚16827.017809584 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35542944073677063 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31334686279296875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14440621435642242 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12281191349029541 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 6validation_average_jet_accuracy‚0.3895169794559479 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚9239 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8124983906745911 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8470970392227173 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13118936121463776 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5057162642478943 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.54706871509552 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8426099419593811 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.804722249507904 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22925376892089844 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4555378556251526 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5990132093429565 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2188001573085785 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06176890805363655 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8465007543563843 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7142072319984436 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 'particle/f_score‚0.6581207513809204 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23970897495746613 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6968285441398621 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30036118626594543 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019327236339449883 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30030062794685364 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04424720257520676 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29927775263786316 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8867693543434143 particle/accuracy_1_of_3‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04361245036125183 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7696545720100403 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07525750249624252 /particle/accuracy_3_of_3‚0.9993678331375122 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3881855309009552 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.736072838306427 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 _step‚768 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2915899455547333 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44566306471824646 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6797268986701965 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6457468271255493 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30036118626594543 +jet/accuracy_2_of_2‚0.16868984699249268 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6639755964279175 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8770803213119507 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt0SRqqt2FBt/FBIpw¾t_purity‚1 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9144653677940369 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3154066503047943 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33176812529563904 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08873599022626877 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6953314542770386 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.018681760877370834 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7344580888748169 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7103403210639954 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7329882979393005 +particle/sensitivity‚0.6440196633338928 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7649431228637695 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49596482515335083 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31387966871261597 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.052204009145498276 *jet/accuracy_1_of_4‚0.9839088320732117~­Ê»:î'µu :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32572105526924133 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8438156843185425 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8471320867538452 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6985534429550171 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.645849347114563 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5885108709335327 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24599085748195648 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7357410788536072 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021752577275037766 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8127322793006897 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3856492042541504 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2441018968820572 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7276562452316284 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01516555342823267 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8758172392845154 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 *jet/accuracy_2_of_3‚0.6007881760597229 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6493884921073914 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8867693543434143 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7998046278953552 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4489903450012207 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01516555342823267 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2579079270362854 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658285915851593 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022750183939933777 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2587502598762512 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8190836906433105 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7487757802009583 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9290577173233032 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 +jet/accuracy_1_of_1‚0.35301175713539124 % _timestamp‚1.7848560636595554e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7577517628669739 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9467796683311462 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 _runtime‚16827.017809584 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35542944073677063 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31334686279296875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14440621435642242 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12281191349029541 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 6validation_average_jet_accuracy‚0.3895169794559479 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚9239 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8124983906745911 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8470970392227173 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13118936121463776 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5057162642478943 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.54706871509552 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8426099419593811 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.804722249507904 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22925376892089844 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4555378556251526 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5990132093429565 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2188001573085785 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06176890805363655 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8465007543563843 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7142072319984436 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 'particle/f_score‚0.6581207513809204 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23970897495746613 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6968285441398621 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30036118626594543 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019327236339449883 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30030062794685364 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04424720257520676 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29927775263786316 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8867693543434143 particle/accuracy_1_of_3‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04361245036125183 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7696545720100403 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07525750249624252 /particle/accuracy_3_of_3‚0.9993678331375122 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3881855309009552 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.736072838306427 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 _step‚768 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2915899455547333 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44566306471824646 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6797268986701965 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6457468271255493 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30036118626594543 +jet/accuracy_2_of_2‚0.16868984699249268 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6639755964279175 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8770803213119507 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9144653677940369 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3154066503047943 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33176812529563904 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08873599022626877 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6953314542770386 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.018681760877370834 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7344580888748169 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7103403210639954 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7329882979393005 +particle/sensitivity‚0.6440196633338928 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7649431228637695 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49596482515335083 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31387966871261597 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.052204009145498276 *jet/accuracy_1_of_4‚0.9839088320732117 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2510862648487091 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9238861203193665 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6239804625511169 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21539664268493652 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6923638582229614 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7142553925514221 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.360604465007782 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24032875895500183 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6493884921073914 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 *jet/accuracy_1_of_2‚0.7711294293403625 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚7.797055892397236 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.624163031578064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12506315112113953 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22889353334903717 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.654193103313446 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6441582441329956 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 epoch‚32 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699317336082458 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.635172426700592 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6088438630104065 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7275477051734924 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8406385183334351 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2703915238380432 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6756872534751892 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9651569724082947 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6777549982070923 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8564395308494568 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11173006147146225 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25109294056892395 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0€p ’Rï'‚(:Ç  ‚üŠÓ¶ýö³ disk.drbd1817.out‚0 disk.drbd1510.in‚0 cpu‚25.335061812599022 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625" disk.dm-0.out‚47081.3662109375 disk.drbd1965.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8597564697266 disk.sda2.out‚ 752.75390625 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51387812149102 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0' proc.memory.availableMB‚ 1931.390625 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 86.6650390625% disk.nvme0n1.out‚47081.3662109375 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2949.94140625 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚12.003342310587566 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 network.sent‚ 1.6204367e+07 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 752.75390625$ memory_percent‚92.14115142822266 network.recv‚ 1.16859e+06 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 101.85546875 proc.cpu.threads‚17†Â–ßžð'‚(:“ ‚üŠÓÙ±Û³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚285.026' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.00866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LµÆR4ñ'/ _wandbruntime‚16842  _runtime‚16842ÿÖ¼šÉò'à _runtime‚16842.876908183 _step‚769 ' lr-AdamW/pg1‚3.1369236047552677e-05 ' lr-AdamW/pg2‚3.1369236047552677e-05 trainer/global_step‚9249 $ _timestamp‚1.784856079519648e+09)îzÎó'È ' lr-AdamW/pg1‚3.1369236047552677e-05 ' lr-AdamW/pg2‚3.1369236047552677e-05 trainer/global_step‚9249 $ _timestamp‚1.784856079519648e+09 _runtime‚16842.876908183 _step‚769„á4ô'/ _wandbruntime‚16842  _runtime‚16842žÎ“­ (õ'„P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9654920118784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80350420 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85919068 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2673088966703521 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2674157618697216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166269488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2664501326968832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2663450238566608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2674156247087616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2673090336742775 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9654920118784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 % _timestamp‚1.7848560795207577e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24652139 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚16842.878311359 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2674157618697216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80350420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24652144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2674156247087616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52848654 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2664501353142272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2663450213138640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740542028558336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _step‚770 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11809596352512 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80350414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28196766 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9640098176167 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85909004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740541625977344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚728732064114688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85919068 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85909004 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2664501326968832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11809561862656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52848910 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740542028558336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_t"$jÅÇrain_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _step‚770 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11809596352512 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80350414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28196766 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9640098176167 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85909004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740541625977344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚728732064114688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85919068 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85909004 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2664501326968832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11809561862656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52848910 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9654920118784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80350420 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85919068 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2673088966703521 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2674157618697216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166269488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2664501326968832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2663450238566608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2674156247087616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2673090336742775 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050‚6œ4÷'/ _wandbruntime‚16844  _runtime‚16844äÔã'ø'ŽN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80358781 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85918086 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9641118977919 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28199763 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2674436178886144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166286951 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80358787 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2664780262432768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166276867 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚728810650480640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36579840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80358781 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85928164 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _step‚771 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80358787 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740621526386176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2663729040340384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404670976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810912485376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85918086 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9657288070144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24654784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28199501 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7848560810006201e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166286951 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11810875905536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚16844.358288278 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85928164 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚728811018571776 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2664780262432768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2673370159318303 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2673368789278937 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639774366521 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52854280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28199763 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2674436178886144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166286951 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80358787 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2664780262432768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166276867 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚728810650480640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36579840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80358781 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85928164 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _step‚771 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80358787 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740621526386176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2663729040340384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404670976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810912485376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85918086 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9657288070144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24654784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28199501 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7848560810006201e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166286951 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11810875905536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚16844.358288278 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85928164 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚728811018571776 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2664780262432768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2673370159318303 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2673368789278937 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639774366521 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52854280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86375020 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80768764 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2688152901168640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2678446817667072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744534605469696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2688154272778240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744534192402944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚732661750503424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24784139 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚774 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9706109675008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28345292 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚16929.76491329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2677390301827440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2687080159275129 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2678446817667072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9707455111168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167133720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167143790 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2678446791493632 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2678446791493632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2688154272778240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80768770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86364956 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53129431 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9706109675008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752«×<†(Ž(ˆP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚332 trainer/global_step‚9299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24784144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9706109675008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80768764 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9707455111168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53129763 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80768770 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚732661382412288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9691202058975 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9689857447689 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86375020 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80768764 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2688152901168640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2678446817667072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744534605469696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2688154272778240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744534192402944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚732661750503424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24784139 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚774 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9706109675008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28345292 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚16929.76491329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2677390301827440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2687080159275129 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2678446817667072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9707455111168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167133720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167143790 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2678446791493632 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2678446791493632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2688154272778240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80768770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86364956 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53129431 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53135038 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2678725700784128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86384116 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24786784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167161253 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11874117507072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9692222860727 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28348259 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677669103601216 _runtime‚16931.246934319 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9707132183040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167151169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80777131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚732739968778240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚9299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚744614454376448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚744614041316864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167161253 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80777137 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 % _timestamp‚1.7848561678893127e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80777131 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _step‚775 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86374038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167151169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86384116 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚23i(P'‘(“N BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677669103601216 _runtime‚16931.246934319 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9707132183040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167151169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80777131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚732739968778240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚9299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚744614454376448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚744614041316864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167161253 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80777137 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 % _timestamp‚1.7848561678893127e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80777131 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _step‚775 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86374038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167151169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86384116 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2678725726957568 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2688434204583936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚332 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2687361351889911 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2687359981850545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚732740336869376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9707132183040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80777137 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11874072538624 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2688434204583936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86374038 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2677669129029184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53135038 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2678725700784128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86384116 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24786784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167161253 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11874117507072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9692222860727 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28348259 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824‡ÄÃ))4’(/ _wandbruntime‚16931  _runtime‚16931͟楓(‚ 0loss/FRt1/assignment_loss‚1.6465735300429998 2loss/SRqqt2/assignment_loss‚0.4339077153825313 2loss/FBt2/assignment_loss‚0.029290817610221887 1loss/SRqqt1/detection_loss‚0.8709389506031877 /loss/FBt2/detection_loss‚0.3013791884581208 2loss/SRqqt1/assignment_loss‚0.4572900545576168 4loss/FBt1/assignment_loss‚4.3143778478010445e-06 /loss/FRt1/detection_loss‚0.5631823926836099 ,loss/FRt2/detection_loss‚1.3844278201534 1loss/SRqqt2/detection_loss‚0.5953521230105115 .loss/FBt1/detection_loss‚0.665724212769419 %loss/total_loss‚7.637910428458031 $ _timestamp‚1.784856167925425e+09 0loss/FRt2/assignment_loss‚0.6898393088085636 epoch‚33 _step‚776 trainer/global_step‚9299 _runtime‚16931.282618787'Íß ”(‡ $ _timestamp‚1.784856167925425e+09 0loss/FRt2/assignment_loss‚0.6898393088085636 epoch‚33 _step‚776 trainer/global_step‚9299 _runtime‚16931.282618787 0loss/FRt1/assignment_loss‚1.6465735300429998 2loss/SRqqt2/assignment_loss‚0.4339077153825313 2loss/FBt2/assignment_loss‚0.029290817610221887 1loss/SRqqt1/detection_loss‚0.8709389506031877 /loss/FBt2/detection_loss‚0.3013791884581208 2loss/SRqqt1/assignment_loss‚0.4572900545576168 4loss/FBt1/assignment_loss‚4.3143778478010445e-06 /loss/FRt1/detection_loss‚0.5631823926836099 ,loss/FRt2/detection_loss‚1.3844278201534 1loss/SRqqt2/detection_loss‚0.5953521230105115 .loss/FBt1/detection_loss‚0.665724212769419 %loss/total_loss‚7.637910428458031ˆíå'û“•(‚(:ˆ ëüŠÓ¹¼³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚181.014' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚60.338 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾¹‚èT–(‚(:É  ëüŠÓ˜’ݳ disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1510.in‚0 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2798.91796875 disk.drbd1931.out‚0# disk.dm-0.out‚47084.02392578125* proc.memory.percent‚11.388826370239258 disk.drbd1285.out‚0 disk.sda.out‚ 755.02734375 disk.sda2.in‚ 0.03515625$ memory_percent‚91.62047704060873 disk.drbd1673.in‚0 cpu‚25.334683411770886 disk.drbd2111.out‚0 disk.drbd1828.in‚0( disk./.usagePercent‚70.5138717213558 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0( proc.memory.availableMB‚ 2059.3515625 network.sent‚ 1.6333386e+07& disk.nvme0n1.out‚47084.02392578125 disk.rbd1.out‚ 87.181640625 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.8596420288086 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 755.02734375 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 network.recv‚ 1.178716e+06 disk.nvme0n1.in‚ 101.85546875 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0´Ùf O—(‚(:Ä  úüŠÓÊÈʳ) proc.memory.availableMB‚ 2059.44140625 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 101.85546875 cpu‚25.334609077721296 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 101.85546875# disk.dm-0.out‚47084.33056640625 disk.drbd1828.in‚0 disk.rbd1.out‚ 87.244140625& disk.nvme0n1.out‚47084.33056640625 disk.drbd1289.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.388826370239258 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1938.out‚0 network.recv‚ 1.181308e+06 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 network.sent‚ 1.6393361e+07% disk./.usageGB‚1260.8596229553223) disk./.usagePercent‚70.51387065466659 proc.cpu.threads‚17 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.drbd1965.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 755.40625 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 755.40625 disk.drbd1226.out‚0 disk.drbd1510.in‚0# memory_percent‚91.6201114654541 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0Ú€ÂÀŸ˜(‚(:” úüŠÓÉܬ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚302.355' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.78500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õàZ¶™(‚(:’ ‰ýŠÓîÂÙ³ gpu.0.gpu‚14.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚297.893' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.29766666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 V)¦Tš(‚(:É  ‰ýŠÓÑö³ disk.drbd1931.in‚0 disk.drbd1289.in‚0 network.sent‚ 1.6396401e+07 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 87.244140625) proc.memory.availableMB‚ 2059.44140625 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1285.in‚0 cpu‚25.33456277122979% disk./.usageGB‚1260.8596229553223 disk.dm-0.in‚ 101.85546875 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1552.out‚0 disk.drbd2111.out‚0 disk.sda.out‚ 755.73046875 disk.drbd1828.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚47084.50244140625 disk.drbd1552.in‚0* proc.memory.percent‚11.388826370239258 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51387065466659 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 network.recv‚ 1.182421e+06 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.sda2.out‚ 755.73046875 disk.rbd1.in‚0# memory_percent‚91.6201114654541 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚47084.50244140625 disk.drbd2041.in‚0±À–ûž›(‚(:“ ˜ýŠÓùÃà³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.712' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.57066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷öoTœ(‚(:É  ˜ýŠÓ¹°ƒ´ network.sent‚ 1.6399402e+07 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.388826370239258 disk.sda.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51387065466659 disk.rbd1.out‚ 87.2763671875 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0& disk.nvme0n1.out‚47084.92041015625 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.in‚0) proc.memory.availableMB‚ 2059.44140625 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1584.in‚0 disk.sda2.out‚ 755.96484375 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0# memory_percent‚91.6201114654541# proc.memory.rssMB‚ 2798.91796875 network.recv‚ 1.18349e+06 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1289.in‚0# disk.dm-0.out‚47084.92041015625 disk.drbd1828.in‚0% disk./.usageGB‚1260.8596229553223 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.sda.out‚ 755.96484375 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 cpu‚25.33450258968388 disk.drbd1552.in‚0 disk.drbd1931.out‚0^C*ž(‚(:“ §ýŠÓÐúæ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.153' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.38433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Ö$Už(‚(:Ê  §ýŠÓ˜‘ˆ´ disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8596801757812 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 756.23828125 cpu‚25.334457394312068 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.sda2.out‚ 756.23828125 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 2059.44140625& disk.nvme0n1.out‚47085.20166015625 disk.drbd1817.in‚0* proc.memory.percent‚11.388826370239258( disk./.usagePercent‚70.5138738547342 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 network.recv‚ 1.184504e+06 disk.drbd1965.in‚0 network.sent‚ 1.6402448e+07 disk.drbd1226.out‚0# disk.dm-0.out‚47085.20166015625 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 101.85546875# memory_percent‚91.6201114654541# proc.memory.rssMB‚ 2798.91796875 proc.cpu.threads‚17 disk.rbd1.out‚ 87.3701171875 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0³þœ“Ÿ(‚(:ˆ ¶ýŠÓî’˳ gpu.0.gpu‚75.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚292.836' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.612 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î"3 Q (‚(:Æ  ¶ýŠÓ®ç³ disk.drbd2078.out‚0 network.recv‚ 1.185506e+06 disk.drbd1938.in‚0$ disk./.usageGB‚1260.859706878662$ disk.nvme0n1.out‚47085.533203125# proc.memory.rssMB‚ 2798.91796875 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 756.68359375 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 2059.27734375 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0! disk.dm-0.out‚47085.533203125 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 756.68359375 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 87.4326171875 disk.drbd1673.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1356.out‚0$ memory_percent‚91.62077903747559) disk./.usagePercent‚70.51387534809909 disk.drbd1584.out‚0 disk.drbd1965.in‚0 cpu‚25.33439853281304 disk.drbd2118.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.85546875 network.sent‚ 1.6405454e+07 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.388826370239258 proc.cpu.threads‚17ûû`Ô4¡(/ _wandbruntime‚17016  _runtime‚17016§OÉ¢(à trainer/global_step‚9349 $ _timestamp‚1.784856253322648e+09 _runtime‚17016.679902439 _step‚777 ' lr-AdamW/pg1‚3.0188815925131566e-05 ' lr-AdamW/pg2‚3.0188815925131566e-05PgïQΣ(È $ _timestamp‚1.784856253322648e+09 _runtime‚17016.679902439 _step‚777 ' lr-AdamW/pg1‚3.0188815925131566e-05 ' lr-AdamW/pg2‚3.0188815925131566e-05 trainer/global_step‚9349‰¦ÊG€4¤(/ _wandbruntime‚17016  _runtime‚17016!}+V¥(ƒP TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9349 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28492302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11935178452992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚778 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2692392282191872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9757299231232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81187114 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736591068801024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24916144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 dQx{øÔ³DeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11935240205824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81187114 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2702149555249664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚372 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24916139 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚367 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚748526309006848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736590700709888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2701071351846737 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2692392256018432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81187120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9742305941783 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168008022 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2692392256018432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2702149555249664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2702150926859264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9758644667392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168018092 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28492669 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9740961330497 _runtime‚17016.681227664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53408813 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53408441 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748525879162880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2701072721885991 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚778 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2692392282191872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9757299231232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81187114 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736591068801024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24916144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11935240205824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81187114 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2702149555249664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚372 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24916139 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚367 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚748526309006848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736590700709888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2701071351846737 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2692392256018432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81187120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9742305941783 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168008022 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2692392256018432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2702149555249664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2702150926859264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9758644667392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168018092 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28492669 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9740961330497 _runtime‚17016.681227664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53408813 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53408441 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748525879162880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2701072721885991 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Šåù14§(/ _wandbruntime‚17018  _runtime‚17018;‘a'¨(N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53413981 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81195481 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86829990 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2691609217717984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168035555 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848562547986045e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9758321739264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11936476981760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736669655166976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168035555 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81195487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2702430858664960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86840068 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28495557 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86840068 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2692671191482368 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11936423624704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81195481 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9743326743535 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9741982132137 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9759667182592 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚779 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2702429487048192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86829990 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2692671165308928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24918784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9759667182592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2692671165308928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168025471 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168025471 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2702430858664960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24918779 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28495202 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚748605710700544r^ÐÕ©(’N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9759667182592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2692671165308928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168025471 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168025471 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2702430858664960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24918779 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28495202 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚748605710700544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53413981 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81195481 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86829990 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2691609217717984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168035555 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848562547986045e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceSt^`CatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9758321739264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11936476981760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736669655166976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168035555 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81195487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2702430858664960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86840068 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28495557 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86840068 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2692671191482368 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11936423624704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81195481 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9743326743535 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9741982132137 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9759667182592 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚779 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2702429487048192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86829990 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2692671165308928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24918784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0‹eKg‚4ª(/ _wandbruntime‚17018  _runtime‚17018–ou «(… % _timestamp‚1.7848562548373508e+09 3loss/FBt1/assignment_loss‚0.0009233851276661323 2loss/FBt2/assignment_loss‚0.028769465638028822 /loss/SRqqt1/detection_loss‚0.95034464628605 %loss/total_loss‚7.605075122232464 epoch‚33 0loss/FRt2/assignment_loss‚0.5765738131351841 /loss/FRt1/detection_loss‚0.5402628288765574 0loss/FBt1/detection_loss‚0.48217222488994255 _runtime‚17018.194550758 _step‚780 2loss/SRqqt1/assignment_loss‚0.4776387810109179 1loss/SRqqt2/detection_loss‚0.5502532736611974 0loss/FRt1/assignment_loss‚1.6088015464085978 3loss/SRqqt2/assignment_loss‚0.49989141855701946 .loss/FRt2/detection_loss‚1.402031891367814 /loss/FBt2/detection_loss‚0.4874118472734887 trainer/global_step‚9349ÎÞ¹2¬(Š 0loss/FRt1/assignment_loss‚1.6088015464085978 3loss/SRqqt2/assignment_loss‚0.49989141855701946 .loss/FRt2/detection_loss‚1.402031891367814 /loss/FBt2/detection_loss‚0.4874118472734887 trainer/global_step‚9349 % _timestamp‚1.7848562548373508e+09 3loss/FBt1/assignment_loss‚0.0009233851276661323 2loss/FBt2/assignment_loss‚0.028769465638028822 /loss/SRqqt1/detection_loss‚0.95034464628605 %loss/total_loss‚7.605075122232464 epoch‚33 0loss/FRt2/assignment_loss‚0.5765738131351841 /loss/FRt1/detection_loss‚0.5402628288765574 0loss/FBt1/detection_loss‚0.48217222488994255 _runtime‚17018.194550758 _step‚780 2loss/SRqqt1/assignment_loss‚0.4776387810109179 1loss/SRqqt2/detection_loss‚0.5502532736611974Œ•CLÄž­(‚(:“ ÅýŠÓ¦ã¨³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.481' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.49366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÙ8R®(‚(:Ç  ÅýŠÓŸÕ̳ disk.drbd1965.out‚0% disk./.usageGB‚1260.8596496582031 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 cpu‚25.334324965660873 disk.drbd2078.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 757.11328125# proc.memory.rssMB‚ 2798.91796875 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 101.85546875! disk.dm-0.out‚47085.865234375 network.sent‚ 1.6408604e+07 disk.drbd1226.in‚0 disk.drbd1226.out‚0$ memory_percent‚91.62117640177408 disk.drbd1938.in‚0( proc.memory.availableMB‚ 2059.1796875 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51387214803148 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 proc.cpu.threads‚17 network.recv‚ 1.186482e+06 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 87.6123046875 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 757.11328125 disk.drbd1289.out‚0$ disk.nvme0n1.out‚47085.865234375 disk.drbd1510.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0* proc.memory.percent‚11.388826370239258âØÅ0ž¯(‚(:“ ÔýŠÓ†¯ë³ gpu.0.gpu‚97.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.302' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.76733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄýiW°(‚(:Ì  ÔýŠÓ´áˆ´ disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 87.6748046875 disk.drbd1510.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 757.32421875 disk.drbd2111.in‚0$ memory_percent‚91.62119229634604 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 2059.17578125# proc.memory.rssMB‚ 2798.91796875 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.sda2.out‚ 757.32421875% disk./.usageGB‚1260.8596305847168 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0) disk./.usagePercent‚70.51387108134227 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 101.85546875& disk.nvme0n1.out‚47086.43701171875 disk.drbd1226.out‚0 disk.drbd1510.out‚0 network.sent‚ 1.6469131e+07 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.388826370239258 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 network.recv‚ 1.189203e+06 disk.drbd1931.in‚0# disk.dm-0.out‚47086.43701171875 disk.drbd1226.in‚0 cpu‚25.334278711406668 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1289.in‚0·ûǽŸ±(‚(:” ãýŠÓµ¾¸³ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.222' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.40733333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+-j·U²(‚(:Ê  ãýŠÓ¥Ô³ disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 757.6484375 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.388826370239258 disk.dm-0.in‚ 101.85546875 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 87.7685546875 disk.drbd1510.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1550.out‚0 network.recv‚ 1.190355e+06% disk./.usageGB‚1260.8596305847168 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2798.91796875 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1828.out‚0 cpu‚25.334235263940432 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.sda.out‚ 757.6484375 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0$ memory_percent‚91.62119229634604) disk./.usagePercent‚70.51387108134227 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0# disk.dm-0.out‚47086.63623046875 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 network.sent‚ 1.6471639e+07 disk.drbd1356.in‚0& disk.nvme0n1.out‚47086.63623046875 disk.drbd1550.in‚0) proc.memory.availableMB‚ 2059.17578125Ö¶šž³(‚(:“ òýŠÓ‚ÿÛ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚285.131' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.04366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã~-¶Q´(‚(:Æ  òýŠÓ––û³ disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 2059.17578125 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 87.8310546875) disk./.usagePercent‚70.51377550598957 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.sda.out‚ 758.1328125 disk.sda2.in‚ 0.03515625! disk.dm-0.out‚47087.169921875 disk.drbd1965.out‚0 disk.drbd1356.in‚0 network.recv‚ 1.191407e+06 disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2798.91796875 disk.drbd1550.out‚0$ memory_percent‚91.62119229634604 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1356.out‚0 cpu‚25.334160442818643 disk.drbd1510.out‚0% disk./.usageGB‚1260.8579216003418 disk.drbd1584.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.85546875$ disk.nvme0n1.out‚47087.169921875 disk.drbd1226.out‚0* proc.memory.percent‚11.388826370239258 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 758.1328125 disk.dm-0.in‚ 101.85546875 disk.drbd1550.in‚0 disk.sda1.out‚0 network.sent‚ 1.6475194e+07 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0µ[¡žµ(‚(:“ þŠÓ¼“ì³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚223.565' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.52166666666666 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V%ulF¶(‚(:»  þŠÓ¼à‡´ disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0% disk./.usageGB‚1260.8579216003418 disk.drbd1552.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 87.8310546875 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 2059.94921875 cpu‚25.334100999402654 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0* proc.memory.percent‚11.388826370239258 proc.cpu.threads‚17 network.recv‚ 1.192542e+06 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 758.765625 disk.dm-0.in‚ 101.85546875 disk.dm-0.out‚ 47087.78125 disk.nvme0n1.in‚ 101.85546875 disk.drbd1289.out‚0) disk./.usagePercent‚70.51377550598957 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.nvme0n1.out‚ 47087.78125 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 758.765625 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 network.sent‚ 1.6477701e+07 disk.drbd2111.in‚0 disk.drbd1673.out‚0# memory_percent‚91.6180451711019D—.}Ÿ·(‚(:” þŠÓ¿Âس gpu.0.gpu‚49.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚326.267' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚108.75566666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø=„ÒN¸(‚(:à  þŠÓœú³ disk.drbd1673.out‚0 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0# disk.dm-0.out‚47088.99658203125 disk.drbd2111.in‚0 disk.drbd1552.in‚0& disk.nvme0n1.out‚47088.99658203125 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.sda.out‚ 759.0078125 disk.sda2.out‚ 759.0078125 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1584.in‚0 cpu‚25.3340714785458 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.sda1.out‚0* proc.memory.percent‚11.388826370239258 disk.sda2.in‚ 0.03515625 network.sent‚ 1.6480704e+07$ disk./.usageGB‚1260.858039855957 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0( proc.memory.availableMB‚ 2059.9453125) disk./.usagePercent‚70.51378211946265 disk.drbd2118.in‚0 network.recv‚ 1.19358e+06 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 87.86328125$ memory_percent‚91.61806106567383 disk.dm-0.in‚ 101.85546875 disk.drbd1828.in‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2798.91796875 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17ßO4¹(/ _wandbruntime‚17103  _runtime‚17103*úõȺ( & lr-AdamW/pg2‚2.960414640712705e-05 trainer/global_step‚9399 % _timestamp‚1.7848563400340202e+09 _runtime‚17103.391281051 _step‚781 & lr-AdamW/pg1‚2.960414640712705e-05”•¦dÍ»(Ç _runtime‚17103.391281051 _step‚781 & lr-AdamW/pg1‚2.960414640712705e-05 & lr-AdamW/pg2‚2.960414640712705e-05 trainer/global_step‚9399 % _timestamp‚1.7848563400340202e+09Hî:4¼(/ _wandbruntime‚17103  _runtime‚17103#½HC (½(ƒP ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9809834223616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87276860 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87276860 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _runtime‚17103.392716623 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53688143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2716147580940288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2716146209330688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752517993190400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81605470 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28640386 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚740520019007488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53688530 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9809834223616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚382 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2706337746716672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2706337720543232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2715062544418345 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2716146209330688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚752517563346432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11997606091776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2716147580940288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2705270504633008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81605470 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53688143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2716147580940288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2716146209330688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752517993190400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81605470 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28640386 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚740520019007488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53688530 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9809834223616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚382 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2706337746716672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2706337720543232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2715062544418345 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2716146209330688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚752517563346432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11997606091776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2716147580940288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2705270504633008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248ŽGï‹i4¿(/ _wandbruntime‚17104  _runtime‚17104¬[`û'À(ŽN JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848563415161955e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚752597813707776 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2705549306406784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53693741 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168909857 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚783 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87285942 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2716426141129216 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2706616629833728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28642962 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81613831 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9810856738816 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚740598973464576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11998778497536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2715342366993761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87296020 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25050779 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2706616656007168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87296020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28643338 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2705549280978816 trainer/global_step‚9399 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9809511295488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚381 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11998840243200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864ýM<9'Á(“N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11998840243200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848563415161955e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚752597813707776 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2705549306406784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53693741 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168909857 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚783 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87285942 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2716426141129216 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2706616629833728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28642962 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81613831 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9810856738816 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚740598973464576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11998778497536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2715342366993761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87296020 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25050779 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2706616656007168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87296020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28643338 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2705549280978816 trainer/global_step‚9399 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9809511295488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚381 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672&X"³4Â(/ _wandbruntime‚17104  _runtime‚17104‡¿ŠDÃ( 2loss/FBt1/assignment_loss‚0.013742618141285709 1loss/SRqqt2/detection_loss‚0.6305957386867209 epoch‚33 /loss/FRt1/assignment_loss‚1.692637813424972 3loss/SRqqt2/assignment_loss‚0.46758083264275097 %loss/total_loss‚8.022268585036127 _step‚784 0loss/FRt2/assignment_loss‚0.7285420085026834 0loss/FRt1/detection_loss‚0.48807575905834005 /loss/FBt1/detection_loss‚0.7625905409351819 /loss/FBt2/detection_loss‚0.4122846080713715 trainer/global_step‚9399 2loss/SRqqt1/assignment_loss‚0.5552927692842663 2loss/FBt2/assignment_loss‚0.005806493112585281 -loss/FRt2/detection_loss‚1.42015904200616 0loss/SRqqt1/detection_loss‚0.844960361169809 # _timestamp‚1.78485634155082e+09 _runtime‚17104.908073862IÖd¦ Ä(† _step‚784 0loss/FRt2/assignment_loss‚0.7285420085026834 0loss/FRt1/detection_loss‚0.48807575905834005 /loss/FBt1/detection_loss‚0.7625905409351819 /loss/FBt2/detection_loss‚0.4122846080713715 trainer/global_step‚9399 2loss/SRqqt1/assignment_loss‚0.5552927692842663 2loss/FBt2/assignment_loss‚0.005806493112585281 -loss/FRt2/detection_loss‚1.42015904200616 0loss/SRqqt1/detection_loss‚0.844960361169809 # _timestamp‚1.78485634155082e+09 _runtime‚17104.908073862 2loss/FBt1/assignment_loss‚0.013742618141285709 1loss/SRqqt2/detection_loss‚0.6305957386867209 epoch‚33 /loss/FRt1/assignment_loss‚1.692637813424972 3loss/SRqqt2/assignment_loss‚0.46758083264275097 %loss/total_loss‚8.022268585036127ãîŠ2GÅ(‚(:¼  ŸþŠÓ·Öç³ disk.sda.in‚ 0.03515625 disk.rbd1.out‚ 88.04296875 disk.drbd1226.in‚0 proc.cpu.threads‚17 network.sent‚ 1.6483742e+07 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.388826370239258 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.sda.out‚759.25 disk.sda2.in‚ 0.03515625 disk.sda2.out‚759.25 disk.drbd1931.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 cpu‚25.333998394439984$ disk./.usageGB‚1260.861946105957) disk./.usagePercent‚70.51400057741166 disk.drbd1510.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.dm-0.in‚ 101.85546875# disk.dm-0.out‚47089.50830078125 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 101.85546875$ memory_percent‚91.61818822224936 disk.drbd1584.out‚0 network.recv‚ 1.194548e+06 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚47089.50830078125( proc.memory.availableMB‚ 2059.9140625 disk.drbd1550.out‚0 disk.drbd1817.in‚0Ô!Ì¥žÆ(‚(:“ ŸþŠÓ‚©Ë³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚271.663' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.55433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù=ÿŸÇ(‚(:” ®þŠÓé“ã³ gpu.0.gpu‚80.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.198' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.06599999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚031Û®È(‚(:Æ  ®þŠÓ™ù´ disk.drbd1550.in‚0$ memory_percent‚91.61717096964517 disk.drbd1550.out‚0( proc.memory.availableMB‚ 2060.1640625 disk.dm-0.in‚ 101.85546875 network.sent‚ 1.6543726e+07% disk./.usageGB‚1260.8619651794434 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 network.recv‚ 1.197301e+06 disk.drbd1584.in‚0 disk.sda2.out‚ 759.69921875 disk.drbd2041.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.sda.out‚ 759.69921875 disk.drbƒ#À#£d1673.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚47089.8037109375 disk.drbd1510.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0* proc.memory.percent‚11.388826370239258 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.drbd2118.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 88.10546875) disk./.usagePercent‚70.51400164410086 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625" disk.dm-0.out‚47089.8037109375 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.rbd1.in‚0 disk.drbd1226.out‚0 cpu‚25.333968041103988“ žÉ(‚(:“ ½þŠÓÈÒ°³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚283.539' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.51299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¸;NQÊ(‚(:Æ  ½þŠÓ“¢Ð³" disk.dm-0.out‚47090.0458984375 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 759.95703125 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 101.85546875# proc.memory.rssMB‚ 2798.91796875) disk./.usagePercent‚70.51400057741166 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 88.10546875 disk.drbd1584.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 2059.8828125 disk.drbd1673.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1550.in‚0$ memory_percent‚91.61831537882486 network.sent‚ 1.6546761e+07$ disk./.usageGB‚1260.861946105957 disk.drbd1584.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd2078.in‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.388826370239258 cpu‚25.333881365006235 network.recv‚ 1.198309e+06 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚47090.0458984375 disk.sda.in‚ 0.03515625 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.sda.out‚ 759.95703125 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0vmí¹žË(‚(:“ ÌþŠÓ˂ɳ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.675' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.22500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.sË+IÌ(‚(:¾  ÌþŠǪ́ç³ disk.drbd2041.in‚0& disk.nvme0n1.out‚47090.49560546875$ memory_percent‚91.61731402079263 disk.drbd1673.in‚0 disk.sda2.out‚760.25 disk.drbd1356.in‚0 disk.rbd1.in‚0 network.sent‚ 1.6550305e+07 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 88.25390625 disk.drbd1226.in‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.388826370239258 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0) proc.memory.availableMB‚ 2060.12890625 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 network.recv‚ 1.199469e+06% disk./.usageGB‚1260.8619689941406 disk.sda.out‚760.25 disk.drbd2111.in‚0 cpu‚25.333822170891253# disk.dm-0.out‚47090.49560546875 disk.drbd1550.out‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1550.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.51400185743871 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0ésCžÍ(‚(:“ ÛþŠÓ£³Ý³ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.318' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.77266666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܶ·oSÎ(‚(:È  ÛþŠÓù¤ÿ³ disk.rbd1.out‚ 88.34765625 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 cpu‚25.333777577998084 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1226.in‚0( disk./.usagePercent‚70.5140884726021 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 network.recv‚ 1.200482e+06 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd2078.in‚0( proc.memory.availableMB‚ 2059.8828125 disk.dm-0.in‚ 101.85546875# disk.dm-0.out‚47090.67529296875 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.sda1.in‚0 disk.sda.out‚ 760.52734375 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0$ memory_percent‚91.61831537882486 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1828.in‚0* proc.memory.percent‚11.388826370239258 proc.cpu.threads‚17 disk.sda2.out‚ 760.52734375 disk.drbd1289.out‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚47090.67529296875 disk.drbd1584.in‚0 disk.drbd1289.in‚0% disk./.usageGB‚1260.8635177612305 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 network.sent‚ 1.6552812e+07 disk.drbd1673.out‚0dñ>óžÏ(‚(:“ êþŠÓµÔݳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.901' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.30033333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÅŸTÐ(‚(:É  êþŠÓëÇÿ³ disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 88.41015625 disk.sda1.in‚0 disk.drbd1584.out‚0( proc.memory.availableMB‚ 2059.8828125 cpu‚25.333704609920996 disk.sda.out‚ 761.05859375 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚47091.58935546875 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1584.in‚0 disk.drbd1356.out‚0 disk.dm-0.in‚ 101.85546875% disk./.usageGB‚1260.8634948730469 disk.drbd1510.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 network.recv‚ 1.201542e+06& disk.nvme0n1.out‚47091.58935546875 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 761.05859375 disk.drbd2111.in‚0 disk.drbd1510.out‚0 network.sent‚ 1.6555907e+07 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1552.in‚0 proc.cpu.threads‚17* proc.memory.percent‚11.388826370239258 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd2118.in‚0) disk./.usagePercent‚70.51408719257505 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0$ memory_percent‚91.61831537882486# proc.memory.rssMB‚ 2798.91796875rÙI4Ñ(/ _wandbruntime‚17190  _runtime‚17190còÍÜÉÒ(à ' lr-AdamW/pg2‚2.9023272929473956e-05 trainer/global_step‚9449 $ _timestamp‚1.784856426954539e+09 _runtime‚17190.311818977 _step‚785 ' lr-AdamW/pg1‚2.9023272929473956e-05s ÷ ÎÓ(È _runtime‚17190.311818977 _step‚785 ' lr-AdamW/pg1‚2.9023272929473956e-05 ' lr-AdamW/pg2‚2.9023272929473956e-05 trainer/global_step‚9449 $ _timestamp‚1.784856426954539e+09‘_Cþ4Ô(/ _wandbruntime‚17190  _runtime‚17190S4Oä(Õ(P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53967842 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚744449337305088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82023820 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87742876 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2730142863411712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2720283185068032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12059670101504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2729053736989953 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82023820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 434038272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784856426955663e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2720283211241472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25180139 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169756626 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87732812 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9859678343680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82023814 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65947136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚786 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12059604154368 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2720283185068032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25180144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756508941459456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 trainer/global_step‚9449 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169766696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 17190.3132626 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9859678343680 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87742876 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2730144235021312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9861023779840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚385 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9843169096113 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87732812 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2720283211241472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28787698 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82023814 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2730142863411712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2720283185068032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12059670101504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2729053736989953 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82023820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 434038272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784856426955663e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2720283211241472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25180139 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169756626 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87732812 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9859678343680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82023814 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65947136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚786 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12059604154368 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2720283185068032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25180144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756508941459456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 trainer/global_step‚9449 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169766696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 17190.3132626 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9859678343680 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87742876 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2730144235021312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9861023779840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚385 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9843169096113 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87732812 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2720283211241472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28787698 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82023814 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53967842 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚744449337305088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82023820 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0’£âp4×(/ _wandbruntime‚17191  _runtime‚171918jrç Ø(N REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82032187 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚744528291762176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87741894 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2720562120531968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚385 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9860700851712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2720562120531968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2719489369667616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82032187 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9862046295040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9845534509151 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169774075 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756588770245120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ Oz#·,1371616768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2729334929604735 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25182779 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚744527923671040 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82032181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚17191.787421218 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2719489395095584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87751972 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 $ _timestamp‚1.784856428429764e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2730422795210240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9844189897753 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87751972 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169774075 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12060908319744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756589200081920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚390 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚787 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2720562094358528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25182784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2730424166827008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87741894 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53973390 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28790221 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53973000 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2730424166827008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82032181 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28790606 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169784159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9862046295040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2729333559565369 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2720562094358528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12060846574080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169784159<ñã'Ù(’N UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9860700851712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2720562120531968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2719489369667616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82032187 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9862046295040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9845534509151 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169774075 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756588770245120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2729334929604735 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25182779 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚744527923671040 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82032181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 _runtime‚17191.787421218 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2719489395095584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87751972 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 $ _timestamp‚1.784856428429764e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2730422795210240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9844189897753 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87751972 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169774075 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12060908319744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756589200081920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚390 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚787 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2720562094358528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25182784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2730424166827008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87741894 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53973390 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28790221 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53973000 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2730424166827008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82032181 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28790606 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169784159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9862046295040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2729333559565369 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2720562094358528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12060846574080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169784159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82032187 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚744528291762176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87741894 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2720562120531968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚385“#PtÃ4Ú(/ _wandbruntime‚17191  _runtime‚17191‘ õk Û(ƒ 0loss/FRt2/assignment_loss‚0.6140785131533191 epoch‚33 2loss/SRqqt2/assignment_loss‚0.4475011463252767 0loss/SRqqt2/detection_loss‚0.595284506985443 0loss/FBt1/detection_loss‚0.37023239668358016 trainer/global_step‚9449 _step‚788 1loss/SRqqt1/assignment_loss‚0.511135996525652 4loss/FBt1/assignment_loss‚1.0702527734490314e-05 /loss/FRt2/detection_loss‚1.3809817411600753 1loss/SRqqt1/detection_loss‚0.8427093300671475 " _timestamp‚1.7848564284682e+09 0loss/FRt1/assignment_loss‚1.6983017791189798 2loss/FBt2/assignment_loss‚0.017223720877971877 /loss/FRt1/detection_loss‚0.5499749761226033 /loss/FBt2/detection_loss‚0.4196631497525354 %loss/total_loss‚7.447097959300319 _runtime‚17191.825389642Þ UiÜ(ˆ 0loss/FRt1/assignment_loss‚1.6983017791189798 2loss/FBt2/assignment_loss‚0.017223720877971877 /loss/FRt1/detection_loss‚0.5499749761226033 /loss/FBt2/detection_loss‚0.4196631497525354 %loss/total_loss‚7.447097959300319 _runtime‚17191.825389642 0loss/FRt2/assignment_loss‚0.6140785131533191 epoch‚33 2loss/SRqqt2/assignment_loss‚0.4475011463252767 0loss/SRqqt2/detection_loss‚0.595284506985443 0loss/FBt1/detection_loss‚0.37023239668358016 trainer/global_step‚9449 _step‚788 1loss/SRqqt1/assignment_loss‚0.511135996525652 4loss/FBt1/assignment_loss‚1.0702527734490314e-05 /loss/FRt2/detection_loss‚1.3809817411600753 1loss/SRqqt1/detection_loss‚0.8427093300671475 " _timestamp‚1.7848564284682e+09”PW²€SÝ(‚(:È  ùþŠÓ¬Ðس disk.drbd2118.out‚0 disk.drbd1673.out‚0 cpu‚25.33366172295797 network.sent‚ 1.6558953e+07 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0( proc.memory.availableMB‚ 2059.8515625 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0# disk.dm-0.out‚47091.84326171875& disk.nvme0n1.out‚47091.84326171875 disk.drbd1673.in‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.388826370239258 network.recv‚ 1.202639e+06 disk.drbd1965.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.sda.out‚ 761.44921875$ memory_percent‚91.61844253540039) disk./.usagePercent‚70.51409977950766 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2798.91796875% disk./.usageGB‚1260.8637199401855 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 88.41015625 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 761.44921875 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 101.85546875 disk.dm-0.in‚ 101.85546875 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0qOû¬žÞ(‚(:“ ùþŠÓÇš»³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚281.878' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.95933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P)q)žß(‚(:“ ˆÿŠÓ¦°ß³ gpu.0.gpu‚49.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.552' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.51733333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äéc Oà(‚(:Ä  ˆÿŠÓ¢áÿ³ disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0$ disk.nvme0n1.out‚47092.115234375 disk.drbd1673.in‚0 network.recv‚ 1.205336e+06 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 network.sent‚ 1.6619079e+07 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 88.4423828125 disk.drbd1285.in‚0% disk./.usageGB‚1260.8637199401855 disk.drbd1817.out‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 761.6171875 disk.drbd2111.in‚0 cpu‚25.33361702884599 proc.cpu.threads‚17) disk./.usagePercent‚70.51409977950766! disk.dm-0.out‚47092.115234375 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2798.91796875* proc.memory.percent‚11.388826370239258 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.sda2.out‚ 761.6171875 disk.drbd2078.out‚0 disk.drbd1550.in‚0$ memory_percent‚91.61844253540039( proc.memory.availableMB‚ 2059.8515625 disk.sda2.in‚ 0.03515625 disk.drbd1510.in‚0 disk.drbd1510.out‚0»i…¡Pá(‚(:Å  —ÿŠÓ´¹Å³ disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2798.91796875 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1931.in‚0 disk.sda.out‚ 761.9765625$ disk.nvme0n1.out‚47092.298828125 disk.drbd1289.in‚0 disk.drbd1965.out‚0( proc.memory.availableMB‚ 2059.8515625 disk.drbd1552.out‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 88.5361328125 network.recv‚ 1.206276e+06 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.sda1.in‚0 disk.drbd1828.out‚0$ memory_percent‚91.61844253540039 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.sda2.out‚ 761.9765625 disk.dm-0.in‚ 101.85546875 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 101.85546875) disk./.usagePercent‚70.51409977950766 disk.drbd2118.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.388826370239258 network.sent‚ 1.6622125e+07 disk.drbd1673.in‚0! disk.dm-0.out‚47092.298828125 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0% disk./.usageGB‚1260.8637199401855 disk.drbd1938.in‚0 cpu‚25.333545801288977›øÙ“â(‚(:ˆ —ÿŠÓüí¨³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚277.587' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.529 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ª¡–Gã(‚(:¼  ¦ÿŠÓ»Öâ³ network.sent‚ 1.6625126e+07 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0$ memory_percent‚91.61844253540039 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 101.85546875 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 88.5986328125 disk.sda1.in‚0 disk.drbd2118.in‚0 cpu‚25.33348705773699 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2798.91796875 disk.sda2.out‚762.25 disk.drbd2041.in‚0 disk.drbd1673.in‚0" disk.dm-0.out‚47092.6337890625 disk.drbd1285.in‚0 disk.sda.out‚762.25 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0( proc.memory.availableMB‚ 2059.8515625 disk.drbd2078.out‚0% disk.nvme0n1.out‚47092.6337890625 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51409977950766 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8637199401855 disk.drbd1356.out‚0 disk.drbd2118.out‚0 network.recv‚ 1.207212e+06 disk.drbd2111.in‚0* proc.memory.percent‚11.3888263702392587/URžä(‚(:“ ¦ÿŠÓž±Â³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚262.522' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.50733333333332 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ž}éžå(‚(:“ µÿŠÓã—¦³ gpu.0.gpu‚10.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.746' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.248666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒzR/Tæ(‚(:É  µÿŠÓë©Ì³ disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 762.67578125* proc.memory.percent‚11.388826370239258 disk.sda1.in‚0 disk.drbd1510.out‚0 disk.drbd1226.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.51409977950766 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 network.sent‚ 1.6628156e+07 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 cpu‚25.33341519190907 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.03515625% disk./.usageGB‚1260.8637199401855 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0) proc.memory.availableMB‚ 2059.84765625 proc.cpu.threads‚17 disk.rbd1.out‚ 88.5986328125 disk.drbd1584.in‚0 disk.dm-0.in‚ 101.85546875$ memory_percent‚91.61845842997232 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2798.91796875 disk.sda1.out‚0 disk.sda.out‚ 762.67578125 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1552.in‚0 disk.drbd1673.in‚0" disk.dm-0.out‚47092.8369140625 disk.drbd2078.out‚0 network.recv‚ 1.208315e+06% disk.nvme0n1.out‚47092.83691406257–Áp4ç(/ _wandbruntime‚17277  _runtime‚17277+ UßÈè( & lr-AdamW/pg1‚2.844627163137668e-05 & lr-AdamW/pg2‚2.844627163137668e-05 trainer/global_step‚9499 % _timestamp‚1.7848565138930957e+09 _runtime‚17277.250329386 _step‚789íãJ–Íé(Ç _runtime‚17277.250329386 _step‚789 & lr-AdamW/pg1‚2.844627163137668e-05 & lr-AdamW/pg2‚2.844627163137668e-05 trainer/global_step‚9499 % _timestamp‚1.7848565138930957e+09•·$¤Ã4ê(/ _wandbruntime‚17277  _runtime‚17277ñ¬´P ë(„P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82442164 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2734228675766272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚760500425131008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2743046299600815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28935139 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25312139 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170640998 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2734228649592832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9912213336064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚790 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88198828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚760500850780672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12121769528320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 % _timestamp‚1.7848565138941398e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54246894 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2744139517492736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82442170 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748378655602688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25312144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9912213336064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9894272978921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82442170 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170630928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2744139517492736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2733150682010608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9895617590207 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224>ìÕ(ì(‰P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2744140889102336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54247283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2733150656582640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88188764 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748379023693824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82442164 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2734228675766272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚760500425131008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2743046299600815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28935139 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25312139 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170640998 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2734228649592832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9912213336064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚790 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88198828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚760500850780672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12121769528320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 % _timestamp‚1.7848565138941398e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54246894 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2744139517492736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82442170 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748378655602688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25312144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9912213336064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9894272978921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82442170 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170630928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2744139517492736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2733150682010608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9895617590207 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82450537 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25314784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9911890407936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170658461 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚17278.725584981 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2734507585056768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848565153679988e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54252875 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9896638391959 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9895293780561 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28937707 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12123067629056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54252486 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28938091 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9913235851264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2743326122176343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82450531 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88207924 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12123007980544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88197846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9911890407936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748457610059776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2734507558883328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82450531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2734507585056768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25314779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚748457241968640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2744419449291264 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88207924 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2744420820908032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768E^¥Ìï(“N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170648377 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2733429483784384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2733429458356416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚17278.725584981 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2734507585056768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848565153679988e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54252875 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9896638391959 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9895293780561 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28937707 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12123067629056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54252486 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28938091 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9913235851264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2743326122176343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82450531 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88207924 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12123007980544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88197846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9911890407936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748457610059776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2734507558883328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82450531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2734507585056768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25314779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚748457241968640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2744419449291264 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88207924 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2744420820908032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170658461 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82450537 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25314784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9911890407936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170658461 F&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8806796669960022 particle/accuracy_1_of_3‚1 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8584055304527283 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8436146974563599 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33171552419662476 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2722487151622772 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 .particle/accuracy_4_of_4‚0.964521050453186 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.706372082233429 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03702365234494209 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8089633584022522 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6986228823661804 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106985211372375 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2611844837665558 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13746747374534607 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05431300774216652 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7242749929428101 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017350273206830025 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14209125936031342 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7848565694296997e+09 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7299866080284119 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8590368032455444 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.261376291513443 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 trainer/global_step‚9519 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6462811827659607 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7303106188774109 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10811556875705719 'particle/accuracy‚0.853875994682312 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6990423202514648 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24001869559288025 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.317239373922348 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6575779318809509 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_3‚0.6056159734725952 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7037805914878845 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 epoch‚33 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0235720407217741 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6768833994865417 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6405993103981018 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6748815774917603 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7681060433387756 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚793 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7212523818016052 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6953433156013489 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.661857545375824 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.045882802456617355 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6529127359390259 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8726643323898315 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3704535663127899 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6362637281417847 particle/accuracy_1_of_1‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7732299566268921 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6522802710533142 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2963370084762573 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3493066430091858 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4601014256477356 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022448912262916565 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2411031872034073 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6282820105552673 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.943337619304657 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5794662237167358 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7506939172744751 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5433968305587769 *jet/accuracy_2_of_4‚0.8447586894035339 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38475993275642395 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8063634634017944 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020753439515829086 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5097824931144714 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8007317781448364 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07052044570446014 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25360870361328125 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 *jet/accuracy_2_of_2‚0.1663292497396469 +jet/accuracy_3_of_4‚0.11347367614507675 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6271820068359375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7357410788536072 particle/accuracy_1_of_4‚1 _runtime‚17332.787655581 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6381378769874573 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3177524507045746 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7653012871742249 7"Purity/2FRt0SRqqt1FBt/event_purity‚0.54358971118927 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 )jet/accuracy_1_of_4‚0.978101909160614 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6816364526748657 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6522802710533142 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4581020474433899 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6485077142715454 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25055354833602905 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22066497802734375 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4998244345188141 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44870617985725403 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02593282423913479 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9121350646018982 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.317239373922348 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 +jet/accuracy_1_of_1‚0.35931095480918884 )loss/val_total_loss‚7.765228506445161 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8738440871238708 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7417588233947754 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3683982193470001 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9207094311714172 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8220599293708801 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.708363950252533 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_1_of_3‚0.9303408265113831 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3869943916797638 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21723893284797668 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3300558626651764 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.39471104741096497 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6002488732337952 *jet/accuracy_1_of_2‚0.7782820463180542 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2392539530992508 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8085107803344727 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33449962735176086 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30865398049354553 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6644189953804016 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32944902777671814 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15561258792877197 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.708030641078949 8"Purity/2FRt0SRqqt*FBt/event_purity‚0.139352485537529 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8806796669960022 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017350273206830025 particle/accuracy_3_of_4‚1 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2735426723957062 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12391798198223114 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8565707802772522 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8553650975227356 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837;þì€T ý(Äu 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6381378769874573 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3177524507045746 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7653012871742249 7"Purity/2FRt0SRqqt1FBt/event_purity‚0.54358971118927 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 )jet/accuracy_1_of_4‚0.978101909160614 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6816364526748657 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6522802710533142 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4581020474433899 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6485077142715454 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25055354833602905 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22066497802734375 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4998244345188141 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44870617985725403 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02593282423913479 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9121350646018982 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.317239373922348 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 +jet/accuracy_1_of_1‚0.35931095480918884 )loss/val_total_loss‚7.765228506445161 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8738440871238708 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7417588233947754 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3683982193470001 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9207094311714172 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8220599293708801 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.708363950252533 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 ?q²!v/&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_1_of_3‚0.9303408265113831 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3869943916797638 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21723893284797668 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3300558626651764 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.39471104741096497 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6002488732337952 *jet/accuracy_1_of_2‚0.7782820463180542 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2392539530992508 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8085107803344727 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33449962735176086 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30865398049354553 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6644189953804016 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32944902777671814 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15561258792877197 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.708030641078949 8"Purity/2FRt0SRqqt*FBt/event_purity‚0.139352485537529 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8806796669960022 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017350273206830025 particle/accuracy_3_of_4‚1 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2735426723957062 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12391798198223114 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8565707802772522 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8553650975227356 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2588183581829071 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444680094718933 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.31734731793403625 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 +jet/accuracy_3_of_3‚0.07535088062286377 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7319027781486511 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2703915238380432 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8806796669960022 particle/accuracy_1_of_3‚1 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8584055304527283 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8436146974563599 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33171552419662476 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2722487151622772 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 .particle/accuracy_4_of_4‚0.964521050453186 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.706372082233429 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03702365234494209 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8089633584022522 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6986228823661804 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106985211372375 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2611844837665558 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13746747374534607 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05431300774216652 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7242749929428101 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017350273206830025 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14209125936031342 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7848565694296997e+09 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7299866080284119 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8590368032455444 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.261376291513443 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 trainer/global_step‚9519 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6462811827659607 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7303106188774109 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10811556875705719 'particle/accuracy‚0.853875994682312 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6990423202514648 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24001869559288025 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.317239373922348 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6575779318809509 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_3‚0.6056159734725952 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7037805914878845 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 epoch‚33 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0235720407217741 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6768833994865417 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6405993103981018 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6748815774917603 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7681060433387756 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚793 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7212523818016052 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6953433156013489 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.661857545375824 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.045882802456617355 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6529127359390259 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8726643323898315 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3704535663127899 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6362637281417847 particle/accuracy_1_of_1‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7732299566268921 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6522802710533142 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2963370084762573 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3493066430091858 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4601014256477356 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022448912262916565 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2411031872034073 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6282820105552673 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.943337619304657 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5794662237167358 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7506939172744751 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5433968305587769 *jet/accuracy_2_of_4‚0.8447586894035339 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38475993275642395 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8063634634017944 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020753439515829086 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5097824931144714 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8007317781448364 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07052044570446014 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25360870361328125 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 *jet/accuracy_2_of_2‚0.1663292497396469 +jet/accuracy_3_of_4‚0.11347367614507675 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6271820068359375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7357410788536072 particle/accuracy_1_of_4‚1 _runtime‚17332.787655581™“#õžþ(‚(:“ €€‹Ó¾û̳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.23733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#­Z9Mÿ(‚(:  €€‹Ó¢Ìê³ disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 764.3671875 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 89.029296875& disk.nvme0n1.out‚47094.99658203125 disk.drbd1226.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd2078.in‚0 disk.drbd1510.in‚0" memory_percent‚92.027219136556 disk.drbd1584.out‚0 network.sent‚ 1.6700242e+07 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2898.89453125 cpu‚25.33600425177995 disk.sda1.out‚0 disk.sda.out‚ 764.3671875 disk.drbd1965.out‚0# disk./.usageGB‚1260.86372756958 disk.drbd1584.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚47094.99658203125 disk.nvme0n1.in‚ 101.85546875 proc.cpu.threads‚17) disk./.usagePercent‚70.51410020618334 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0' proc.memory.availableMB‚ 1959.390625 network.recv‚ 1.215253e+06 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.795632044474285 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0S¹î§ž€)‚(:“ €‹Óµ¥µ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚269.579' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.85966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06d›úT)‚(:É  €‹Ó¦ƒÕ³ disk.drbd1289.out‚0 disk.drbd1510.in‚0$ memory_percent‚92.02729860941568 network.sent‚ 1.6754134e+07 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2898.89453125 proc.cpu.threads‚17# disk.dm-0.out‚47095.17626953125 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0) proc.memory.availableMB‚ 1959.37109375 disk.drbd1356.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚47095.17626953125 disk.drbd1552.in‚0) disk./.usagePercent‚70.51410020618334 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 101.85546875 cpu‚25.335944948641167 disk.drbd1673.in‚0 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.795632044474285 disk.drbd1965.in‚0# disk./.usageGB‚1260.86372756958 disk.sda.out‚ 764.68359375 disk.drbd1828.out‚0 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 101.85546875 disk.drbd1356.out‚0 disk.drbd1965.out‚0 network.recv‚ 1.217955e+06 disk.drbd1828.in‚0 disk.rbd1.out‚ 89.123046875 disk.drbd1817.out‚0 disk.sda2.out‚ 764.68359375(‹þêG‚)‚(:¼  ž€‹ÓÊ ä³# disk.nvme0n1.out‚47095.79296875 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 101.85546875 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1584.in‚0 disk.drbd1673.in‚0$ memory_percent‚92.02731450398764 proc.cpu.threads‚17 disk.nvme0n1.in‚ 101.85546875 disk.drbd2118.out‚0 disk.dm-0.out‚47095.79296875( proc.memory.availableMB‚ 1959.3671875 disk.drbd1285.out‚0 disk.sda.out‚ 765.03125 disk.drbd2041.out‚0 disk.rbd1.out‚ 89.185546875 cpu‚25.335899238459167 network.sent‚ 1.6757678e+07 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.795632044474285 network.recv‚ 1.219034e+06 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.sda2.out‚ 765.03125) disk./.usagePercent‚70.51410020618334 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2898.89453125# disk./.usageGB‚1260.86372756958 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚01·:žƒ)‚(:“ ž€‹Ó·ê¾³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.359' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.45299999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÝÝÙ4„)/ _wandbruntime‚17383  _runtime‚17383q_Ç‚Ê…)Ä _runtime‚17383.283307654 _step‚794 ' lr-AdamW/pg1‚2.7873218144485693e-05 ' lr-AdamW/pg2‚2.7873218144485693e-05 trainer/global_step‚9549 % _timestamp‚1.7848566199260874e+09Ò­EÅφ)É ' lr-AdamW/pg1‚2.7873218144485693e-05 ' lr-AdamW/pg2‚2.7873218144485693e-05 trainer/global_step‚9549 % _timestamp‚1.7848566199260874e+09 _runtime‚17383.283307654 _step‚794šÏï\4‡)/ _wandbruntime‚17383  _runtime‚17383oWŒ (ˆ)„P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88692023 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54560260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29109229 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9969021328896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171652039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25451031 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82949952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82949946 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2750734817015808 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82949952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2760703838344704 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9969021328896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88692023 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54560580 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82949946 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚315 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9950985701038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚17383.284632687 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2750734790842368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2760702466735104 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2750734790842368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚764548933027840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚752356973413376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9952330312324 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9967675892736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171641969 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2760703838344704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25451036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2749649746422608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2759600732123646 % _timestamp‚1.7848566199270992e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 410969600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12191591523328 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2759602102162900 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0°y;=b‰)‰P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9969021328896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171652039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25451031 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82949952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82949946 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2750734817015808 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82949952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2760703838344704 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9969021328896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88692023 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54560580 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82949946 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚315 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9950985701038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 _runtime‚17383.284632687 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ]HDeviceStatsç»ÇÛ­Monitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2750734790842368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2760702466735104 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2750734790842368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚764548933027840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚752356973413376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9952330312324 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9967675892736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171641969 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2760703838344704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25451036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2749649746422608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2759600732123646 % _timestamp‚1.7848566199270992e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 410969600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12191591523328 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2759602102162900 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88692023 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54560260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29109229›in 4Š)/ _wandbruntime‚17384  _runtime‚17384NŠ`í'‹)ŽN I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9968698400768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25453671 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752435927870464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82958319 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2751013700132864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54565825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25453676 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29112154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚325 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2760982398533632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171669502 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54566150 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚764628774018048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2760983770150400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2749928573624352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88711183 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2751013700132864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12192799081984 _runtime‚17384.758108489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82958319 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88701105 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88711183 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9970043844096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12192846147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _step‚796 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2760983770150400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752435559779328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2759881924738428 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82958313 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9953351114076 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82958313 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9953351114076 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752435927870464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82958319 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2751013700132864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54565825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25453676 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29112154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚325 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2760982398533632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171669502 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54566150 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚764628774018048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2760983770150400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2749928573624352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88711183 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2751013700132864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12192799081984 _runtime‚17384.758108489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82958319 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88701105 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88711183 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9970043844096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12192846147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _step‚796 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2760983770150400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752435559779328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2759881924738428 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QáŽ)‚ 0loss/FRt2/assignment_loss‚0.5249935361776459 2loss/FBt1/assignment_loss‚7.30496588058966e-05 /loss/FRt2/detection_loss‚1.3892664400033987 0loss/SRqqt2/detection_loss‚0.667316261787412 /loss/FBt2/detection_loss‚0.4705311871651655 %loss/total_loss‚8.315771655951368 epoch‚34 % _timestamp‚1.7848566214398534e+09 3loss/SRqqt2/assignment_loss‚0.47381422134877355 trainer/global_step‚9549 _runtime‚17384.797018909 /loss/FRt1/assignment_loss‚1.576927371243307 3loss/SRqqt1/assignment_loss‚0.49506069926892604 1loss/SRqqt1/detection_loss‚0.8278804093551964 .loss/FBt1/detection_loss‚1.365575368813653 _step‚797 2loss/FBt2/assignment_loss‚0.009138922291543703 -loss/FRt1/detection_loss‚0.51519418883754j%#1 )‡ 2loss/FBt2/assignment_loss‚0.009138922291543703 -loss/FRt1/detection_loss‚0.51519418883754 0loss/FRt2/assignment_loss‚0.5249935361776459 2loss/FBt1/assignment_loss‚7.30496588058966e-05 /loss/FRt2/detection_loss‚1.3892664400033987 0loss/SRqqt2/detection_loss‚0.667316261787412 /loss/FBt2/detection_loss‚0.4705311871651655 %loss/total_loss‚8.315771655951368 epoch‚34 % _timestamp‚1.7848566214398534e+09 3loss/SRqqt2/assignment_loss‚0.47381422134877355 trainer/global_step‚9549 _runtime‚17384.797018909 /loss/FRt1/assignment_loss‚1.576927371243307 3loss/SRqqt1/assignment_loss‚0.49506069926892604 1loss/SRqqt1/detection_loss‚0.8278804093551964 .loss/FBt1/detection_loss‚1.365575368813653 _step‚797¶hèƒJ)‚(:¿  ­€‹Óãˆí³ disk.dm-0.out‚ 47095.984375 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2898.89453125 disk.dm-0.in‚ 101.85546875 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1828.out‚0 disk.rbd1.in‚0* proc.memory.percent‚11.795632044474285! disk.nvme0n1.out‚ 47095.984375 disk.drbd2078.out‚0$ memory_percent‚92.02723503112793 disk.drbd1965.out‚0 disk.rbd1.out‚ 89.185546875 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0# disk./.usageGB‚1260.86372756958 disk.drbd2118.in‚0 disk.drbd1289.out‚0 cpu‚25.335825064401288 disk.sda2.out‚ 765.55859375) proc.memory.availableMB‚ 1959.38671875 network.recv‚ 1.220051e+06 disk.drbd1356.in‚0 disk.sda.out‚ 765.55859375) disk./.usagePercent‚70.51410020618334 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 network.sent‚ 1.6760184e+07 disk.drbd1285.in‚0 disk.drbd1817.out‚0(äŒGž‘)‚(:“ ­€‹Ó¸ùɳ gpu.0.gpu‚36.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.619' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.20633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”×r·’)‚(:’ ¼€‹Ó—°µ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚281.47' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.82333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅοS“)‚(:È  ¼€‹Ó…óѳ disk.dm-0.in‚ 101.85546875 disk.drbd1584.out‚0# memory_percent‚92.0280933380127 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.rbd1.in‚0 network.sent‚ 1.6820182e+07 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.sda.in‚ 0.03515625 network.recv‚ 1.222728e+06 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 101.85546875) disk./.usagePercent‚70.51410020618334 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 89.2177734375 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 765.82421875) proc.memory.availableMB‚ 1959.17578125* proc.memory.percent‚11.795632044474285 cpu‚25.33576619994256 disk.drbd2041.in‚0& disk.nvme0n1.out‚47096.50341796875 disk.drbd1938.out‚0# disk.dm-0.out‚47096.50341796875 disk.drbd1285.in‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17# disk./.usageGB‚1260.86372756958 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 765.82421875 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1584.in‚0ÖµÏ}”)‚(:“ Ë€‹Ó¥¥Ë³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.763' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.58766666666666 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# ‘ÐQ!gpu.0.uncorrectedMemoryErrors‚0|4R•)‚(:Ç  Ë€‹ÓÛáí³ disk.drbd1226.out‚0 disk.sda.out‚ 765.9921875 disk.drbd1938.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1828.in‚0# disk./.usageGB‚1260.86372756958& disk.nvme0n1.out‚47096.68310546875 disk.sda2.out‚ 765.9921875 cpu‚25.33572046872926 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51410020618334 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0$ memory_percent‚92.02815691630045 disk.drbd1965.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 101.85546875# disk.dm-0.out‚47096.68310546875 disk.drbd1965.out‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd1510.out‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1959.16015625 disk.rbd1.out‚ 89.3115234375 disk.nvme0n1.in‚ 101.85546875 disk.drbd2078.in‚0 network.recv‚ 1.223756e+06 disk.drbd1673.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.795632044474285 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 network.sent‚ 1.6823722e+07 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0’¢S–)‚(:È  Ú€‹ÓÀ‴ disk.drbd2041.out‚0 disk.nvme0n1.in‚ 101.85546875 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.02879269917807* proc.memory.percent‚11.795632044474285 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51389540185613 disk.rbd1.out‚ 89.3115234375& disk.nvme0n1.out‚47097.01904296875 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda.out‚ 766.4453125 disk.sda2.out‚ 766.4453125 cpu‚25.33564639422993$ disk./.usageGB‚1260.860065460205 disk.drbd1828.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.in‚0 network.sent‚ 1.6826227e+07 network.recv‚ 1.224899e+06 disk.drbd1285.in‚0 disk.dm-0.in‚ 101.85546875 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0# disk.dm-0.out‚47097.01904296875) proc.memory.availableMB‚ 1959.00390625Ã}-îž—)‚(:“ Ú€‹Ó­‰ã³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.413' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.13766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀÉ¥ž˜)‚(:“ 逋Ó¤ëγ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.084' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.02799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jq N™)‚(:à  逋Óå¸í³ network.sent‚ 1.682923e+07 disk.drbd1226.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 766.77734375 disk.drbd1965.in‚0 disk.sda.out‚ 766.77734375 disk.sda2.in‚ 0.03515625* proc.memory.percent‚11.795632044474285 network.recv‚ 1.225842e+06 disk.drbd1550.in‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2898.89453125 disk.rbd1.out‚ 89.4599609375& disk.nvme0n1.out‚47097.19482421875 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0$ memory_percent‚92.02898343404135) proc.memory.availableMB‚ 1958.95703125 disk.dm-0.in‚ 103.59375 disk.drbd1817.out‚0 disk.drbd1931.out‚0# disk.dm-0.out‚47097.19482421875 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0$ disk./.usageGB‚1260.860065460205 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.out‚0 cpu‚25.33558764742764 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51389540185613 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 تŸš)‚(:” ø€‹ÓŸ™µ³ gpu.0.gpu‚68.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.499' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.49966666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CªP›)‚(:Å  ø€‹Óñ¬Ø³ network.recv‚ 1.226945e+06 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 cpu‚25.335571928627495 network.sent‚ 1.6832234e+07% disk./.usageGB‚1260.8600730895996 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.rbd1.out‚ 89.4599609375* proc.memory.percent‚11.795632044474285 disk.sda2.in‚ 0.03515625 disk.drbd1550.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0$ memory_percent‚92.02909469604492 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51389582853182 disk.sda1.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda.out‚ 767.16796875 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 767.16796875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 103.59375# proc.memory.rssMB‚ 2898.89453125 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0# disk.dm-0.out‚47098.37353515625 disk.rbd1.in‚0& disk.nvme0n1.out‚47098.37353515625 disk.drbd1226.out‚0( proc.memory.availableMB‚ 1958.9296875 disk.drbd1550.in‚0=ˆá4œ)/ _wandbruntime‚17470  _runtime‚174706¬2Ç)Á _step‚798 & lr-AdamW/pg1‚2.730418758298378e-05 & lr-AdamW/pg2‚2.730418758298378e-05 trainer/global_step‚9599 $ _timestamp‚1.784856706777738e+09 _runtime‚17470.134986339‚ìU<Ìž)Æ _runtime‚17470.134986339 _step‚798 & lr-AdamW/pg1‚2.730418758298378e-05 & lr-AdamW/pg2‚2.730418758298378e-05 trainer/global_step‚9599 $ _timestamp‚1.784856706777738e+09žÉºd4Ÿ)/ _wandbruntime‚17470  _runtime‚17470îì-p( )‰P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2774699120816128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29256342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10002089583846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2774699120816128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10018865448960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10018865448960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25583036 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768540347798528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12254056087552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756286291710976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 $ _timestamp‚1.784856706778854e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89147975 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54839720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83368302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83368296 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89147975 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2763589835111408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2764680255367168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89158039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29256684 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚799 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2764680281540608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83368296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25583031 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚756286659802112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10020210885120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12254111548928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2763589860539376 trainer/global_step‚9599 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172526341 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89158039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2773593294734508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10020210885120 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83368302 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚17470.136441704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2774700492425728 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10003434195132 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89158039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2773593294734508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10020210885120 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83368302 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚17470.136441704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2774700492425728 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10003434195132 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2774699120816128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10018865448960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10018865448960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25583036 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768540347798528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12254056087552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756286291710976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 $ _timestamp‚1.784856706778854e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89147975 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54839720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83368302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83368296 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89147975 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2763589835111408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2764680255367168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89158039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29256684 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _step‚799 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2764680281540608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83368296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25583031 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚756286659802112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ŸL4¢)/ _wandbruntime‚17471  _runtime‚17471Ô‡+Jÿ£)”N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172543804 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10003110385486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768620172618752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2774980424231424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89157057 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768620594066944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2763868636885184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚17471.614567847 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10019887956992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚756364878076928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9599 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83376669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12255347898880 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54844904 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10019887956992 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2763868662313152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2774979052614656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚756365246168064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83376663 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2764959190831104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976ç°5'¤)™N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768620172618752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2774980424231424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89157057 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768620594066944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2763868636885184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚17471.614567847 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10019887956992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚756364878076928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9599 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83376669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12255347898880 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚345 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54844904 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10019887956992 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2763868662313152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2774979052614656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚756365246168064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83376663 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2764959190831104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172543804 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10003110385486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 vÇ54¥)/ _wandbruntime‚17471  _runtime‚17471 t×f ¦)„ 1loss/FBt2/assignment_loss‚0.02365008702139911 0loss/FRt1/assignment_loss‚1.5697561571295833 /loss/FRt2/detection_loss‚1.4783096792445776 0loss/FRt2/assignment_loss‚0.6294194867188794 2loss/SRqqt2/assignment_loss‚0.4729473122513685 /loss/FRt1/detection_loss‚0.5234444129977923 1loss/SRqqt1/detection_loss‚0.9226107073490531 0loss/SRqqt2/detection_loss‚0.664044543553987 /loss/FBt1/detection_loss‚0.5656494449304287 0loss/FBt2/detection_loss‚0.40686755221774035 trainer/global_step‚9599 2loss/SRqqt1/assignment_loss‚0.4854921750532918 4loss/FBt1/assignment_loss‚0.00026095939121791067 $loss/total_loss‚7.74245251785932 epoch‚34 $ _timestamp‚1.784856708292876e+09 _runtime‚17471.650122093 _step‚801õö^ͧ)‰ $loss/total_loss‚7.74245251785932 epoch‚34 $ _timestamp‚1.784856708292876e+09 _runtime‚17471.650122093 _step‚801 1loss/FBt2/assignment_loss‚0.02365008702139911 0loss/FRt1/assignment_loss‚1.5697561571295833 /loss/FRt2/detection_loss‚1.4783096792445776 0loss/FRt2/assignment_loss‚0.6294194867188794 2loss/SRqqt2/assignment_loss‚0.4729473122513685 /loss/FRt1/detection_loss‚0.5234444129977923 1loss/SRqqt1/detection_loss‚0.9226107073490531 0loss/SRqqt2/detection_loss‚0.664044543553987 /loss/FBt1/detection_loss‚0.5656494449304287 0loss/FBt2/detection_loss‚0.40686755221774035 trainer/global_step‚9599 2loss/SRqqt1/assignment_loss‚0.4854921750532918 4loss/FBt1/assignment_loss‚0.00026095939121791067¡Í«»OM¨)‚(:  ‡‹ÓþÉ´ cpu‚25.335483168780495 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0) proc.memory.availableMB‚ 1959.10546875 proc.cpu.threads‚17 network.recv‚ 1.22799e+06) disk./.usagePercent‚70.51392420246464 disk.drbd1552.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 767.42578125 disk.dm-0.in‚ 103.59375" disk.dm-0.out‚47098.9462890625* proc.memory.percent‚11.795632044474285 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 89.7021484375 disk.drbd1510.out‚0 network.sent‚ 1.6835823e+07$ memory_percent‚92.02837944030762 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0$ disk./.usageGB‚1260.860580444336 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.out‚0% disk.nvme0n1.out‚47098.9462890625 disk.sda2.out‚ 767.42578125 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1226.out‚0 disk.drbd1550.out‚0Zsºž©)‚(:“ ‡‹ÓêÜÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚284.317' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.77233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F‹y@ª)‚(:µ  –‹ÓÌÙ€´ network.recv‚ 1.230547e+06 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1289.out‚0' proc.memory.availableMB‚ 1958.640625% disk./.usageGB‚1260.8592491149902 disk.drbd1828.in‚0 disk.dm-0.out‚ 47099.53125 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.sda1.out‚0* proc.memory.percent‚11.795632044474285) disk./.usagePercent‚70.51384974755818 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.sda2.out‚ 767.7421875 cpu‚25.335438746704888 disk.drbd1510.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 767.7421875 disk.dm-0.in‚ 103.59375# memory_percent‚92.0302708943685 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 network.sent‚ 1.6895306e+07 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 89.7021484375 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.out‚ 47099.53125 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0@WE«)‚(:’ –‹Óлâ³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.07666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùד®Ÿ¬)‚(:” ¥‹Óã·Ú³ gpu.0.gpu‚75.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.067' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.68900000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·æ’;K­)‚(:À  ¥‹Ó¦˜ú³% disk./.usageGB‚1260.8592491149902 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 103.59375 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1958.87890625 disk.drbd2118.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.795632044474285 disk.drbd1938.in‚0 disk.drbd1965.in‚0 cpu‚25.335394328586304 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.sda2.out‚ 768.11328125 proc.cpu.threads‚17 network.sent‚ 1.6898307e+07 disk.drbd1584.in‚0 disk.sda.out‚ 768.11328125 disk.drbd1938.out‚0 disk.dm-0.out‚47099.72265625 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 103.59375 disk.sda2.in‚ 0.03515625$ memory_percent‚92.02930132548013 network.recv‚ 1.231452e+06 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 89.7646484375# disk.nvme0n1.out‚47099.72265625 disk.drbd1226.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51384974755818’.I²O®)‚(:Ä  ´‹Óíâê³ disk.drbd1226.in‚0 disk.drbd1226.out‚0# disk.dm-0.out‚47100.11572265625 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd1938.out‚0 disk.drbd1965.out‚0 network.recv‚ 1.232624e+06 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.nvme0n1.in‚ 103.59375 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1958.62890625* proc.memory.percent‚11.795632044474285 disk.drbd1828.out‚0 disk.rbd1.out‚ 89.7646484375 disk.drbd1584.out‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51385081424739 disk.drbd1552.out‚0 disk.sda.out‚ 768.3828125 disk.drbd1938.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚47100.11572265625 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0$ memory_percent‚92.03031857808432 cpu‚25.335321724937298 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 768.3828125 disk.drbd1817.out‚0 network.sent‚ 1.6901438e+07 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1828.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8592681884766 disk.sda2.in‚ 0.03515625 disk.drbd2041.in‚0GP—ž¯)‚(:“ ´‹Ó»Íʳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.192' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.06400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ío1E°)‚(:º  ËÓ£Êê³ disk.rbd1.out‚ 89.890625 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 103.59375# proc.memory.rssMB‚ 2898.89453125 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 network.sent‚ 1.690444e+07 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0# disk.dm-0.out‚47100.34228515625 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0$ memory_percent‚92.03031857808432 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 cpu‚25.335277336468014 network.recv‚ 1.233608e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 103.59375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 768.65625 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚47100.34228515625 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0* proc.memory.percent‚11.795632044474285 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1958.62890625 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 768.65625% disk./.usageGB‚1260.8602027893066 disk.drbd1817.in‚0( disk./.usagePercent‚70.5139030820184_zþ£Ÿ±)‚(:” ËӼͳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚301.105' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.36833333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Ågž²)‚(:“ Ò‹Óƻdz gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.254' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.08466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨íFH³)‚(:½  Ò‹Ó¯¹ä³( disk./.usagePercent‚70.5139020153292 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 cpu‚25.335218953472957 disk.sda.out‚ 768.97265625 network.recv‚ 1.234698e+06 disk.rbd1.out‚ 89.890625 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1958.62890625 disk.drbd1552.in‚0 disk.drbd1584.in‚0! disk.dm-0.out‚47100.685546875 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2898.89453125$ disk.nvme0n1.out‚47100.685546875 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8601837158203 disk.drbd1510.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 network.sent‚ 1.6907443e+07 disk.drbd1285.out‚0 disk.sda2.out‚ 768.97265625* proc.memory.percent‚11.795632044474285 disk.nvme0n1.in‚ 103.59375 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 103.59375 disk.sda1.out‚0$ memory_percent‚92.03031857808432 proc.cpu.threads‚17 «´4´)/ _wandbruntime‚17556  _runtime‚17556òB™Èµ) & lr-AdamW/pg1‚2.673925453374043e-05 & lr-AdamW/pg2‚2.673925453374043e-05 trainer/global_step‚9649 % _timestamp‚1.7848567933485193e+09 _runtime‚17556.705804751 _step‚802°¬˜Í¶)Ç & lr-AdamW/pg1‚2.673925453374043e-05 & lr-AdamW/pg2‚2.673925453374043e-05 trainer/global_step‚9649 % _timestamp‚1.7848567933485193e+09 _runtime‚17556.705804751 _step‚802¢ð™14·)/ _wandbruntime‚17556  _runtime‚17556"'J!y¸)ŠP \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10054538077940 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2777529949228176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2788695774897152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7848567933496435e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89613991 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚760215610008576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10071400441344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10070055005184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2788697146506752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83786646 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89603927 F=DeviceStatsMonitor.on_train_batch_start/segmentHJ± —.all.allocated‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55118258 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2778625746065408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2788697146506752 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2777529923800208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10070055005184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12316480123392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89603927 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55118619 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2787584487306116 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10053193466654 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2778625719891968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2778625719891968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29403583 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚9649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83786652 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89613991 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2788695774897152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12316428856320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2787583117266862 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10071400441344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚772532458223104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173390573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173400643 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2778625746065408 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚803 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83786652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25715036 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83786646 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755Å÷…(¹)P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29403583 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚9649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83786652 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89613991 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2788695774897152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12316428856320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2787583117266862 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10071400441344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚772532458223104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173390573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173400643 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2778625746065408 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚803 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83786652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25715036 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83786646 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10054538077940 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2777529949228176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2788695774897152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7848567933496435e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89613991 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚760215610008576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10071400441344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10070055005184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2788697146506752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83786646 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89603927 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55118258 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2778625746065408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2788697146506752 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2777529923800208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10070055005184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12316480123392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89603927 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55118619 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2787584487306116 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10053193466654 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2778625719891968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2778625719891968£ÇŒ‹¶4º)/ _wandbruntime‚17558  _runtime‚17558áRΓ'»)“N =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚9649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2778904655355904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12317677223936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173418106 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12317734775296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10054214268294 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2788975706695680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25717671 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚17558.184220212 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2787862939842278 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83795019 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89623087 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89613009 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55123824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 $ _timestamp‚1.784856794826646e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83795019 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2787864309881644 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚760294564465664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173408022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚359 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2777808725573984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173418106 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83795013 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29406507 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83795013 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25717676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29406153³Š©ä¼)˜N PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10054214268294 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2788975706695680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25717671 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚17558.184220212 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2787862939842278 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83795019 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89623087 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89613009 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55123824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 $ _timestamp‚1.784856794826646e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83795019 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2787864309881644 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚760294564465664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173408022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 =†µôE:7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚359 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2777808725573984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173418106 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83795013 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29406507 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83795013 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25717676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29406153 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 trainer/global_step‚9649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2778904655355904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12317677223936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173418106 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12317734775296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0¤ý\ ­4½)/ _wandbruntime‚17558  _runtime‚17558zŒa ¾)… 0loss/FRt1/assignment_loss‚1.6525795814321436 0loss/FBt2/assignment_loss‚0.0204646732251296 %loss/total_loss‚7.554908461383862 % _timestamp‚1.7848567948613157e+09 _runtime‚17558.218509896 0loss/FRt2/assignment_loss‚0.5660776869750317 2loss/SRqqt1/assignment_loss‚0.5675457626850009 3loss/FBt1/assignment_loss‚5.987677323604501e-05 /loss/FRt2/detection_loss‚1.5233220366111462 trainer/global_step‚9649 2loss/SRqqt2/assignment_loss‚0.5243173147533302 /loss/FRt1/detection_loss‚0.5349795413539309 /loss/FBt1/detection_loss‚0.3478299438720569 0loss/FBt2/detection_loss‚0.36974699112729437 epoch‚34 _step‚805 1loss/SRqqt1/detection_loss‚0.8661847798565876 1loss/SRqqt2/detection_loss‚0.5818002727189744OC鬿)Š epoch‚34 _step‚805 1loss/SRqqt1/detection_loss‚0.8661847798565876 1loss/SRqqt2/detection_loss‚0.5818002727189744 0loss/FRt1/assignment_loss‚1.6525795814321436 0loss/FBt2/assignment_loss‚0.0204646732251296 %loss/total_loss‚7.554908461383862 % _timestamp‚1.7848567948613157e+09 _runtime‚17558.218509896 0loss/FRt2/assignment_loss‚0.5660776869750317 2loss/SRqqt1/assignment_loss‚0.5675457626850009 3loss/FBt1/assignment_loss‚5.987677323604501e-05 /loss/FRt2/detection_loss‚1.5233220366111462 trainer/global_step‚9649 2loss/SRqqt2/assignment_loss‚0.5243173147533302 /loss/FRt1/detection_loss‚0.5349795413539309 /loss/FBt1/detection_loss‚0.3478299438720569 0loss/FBt2/detection_loss‚0.36974699112729437¥p~ÁÚGÀ)‚(:¼  á‹ÓäÞò³ disk.sda2.in‚ 0.03515625 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 network.sent‚ 1.6910477e+07 network.recv‚ 1.235652e+06 disk.sda1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.795632044474285 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 103.59375 disk.rbd1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0! disk.dm-0.out‚47101.072265625 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51392228242408 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 103.59375$ memory_percent‚92.02760060628256% disk./.usageGB‚1260.8605461120605 disk.drbd1552.out‚0 disk.sda2.out‚ 769.44140625 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 769.44140625$ disk.nvme0n1.out‚47101.072265625 cpu‚25.335188599667774 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0' proc.memory.availableMB‚ 1959.296875 disk.drbd1289.out‚0 disk.rbd1.out‚ 89.953125 disk.drbd1550.out‚0þÎëúžÁ)‚(:“ á‹Óš¥Ñ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚279.574' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.19133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`*êŸÂ)‚(:” ð‹Ó³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚159.287' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚53.095666666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0° ÉÂIÃ)‚(:¾  ð‹Óá‚ݳ network.sent‚ 1.6970501e+07 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda.in‚ 0.03515625 network.recv‚ 1.238433e+06 disk.drbd1584.out‚0 cpu‚25.335130754008674 disk.drbd2041.out‚0 disk.sda.out‚ 769.76953125 disk.sda2.out‚ 769.76953125 disk.drbd1965.in‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd2041.in‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚47101.3994140625 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625* proc.memory.percent‚11.795632044474285 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0" disk.dm-0.out‚47101.3994140625 disk.drbd1552.in‚0% disk./.usageGB‚1260.8605461120605 disk.dm-0.in‚ 103.59375) disk./.usagePercent‚70.51392228242408 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 89.953125 disk.drbd2078.in‚0$ memory_percent‚92.02760060628256 disk.drbd1356.out‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 1959.296875 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0s¨ºƒŸÄ)‚(:” ÿ‹ÓËóų gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚304.941' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.64699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$õèÂMÅ)‚(:  ÿ‹Ó¬ªç³) disk./.usagePercent‚70.51391801566726 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0$ memory_percent‚92.02659924825033 disk.rbd1.out‚ 90.0791015625 disk.drbd1584.out‚0 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2898.89453125 network.recv‚ 1.239468e+06% disk./.usageGB‚1260.8604698181152 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 769.9921875 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 cpu‚25.335058079501675 proc.cpu.threads‚17 network.sent‚ 1.6973493e+07 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0% disk.nvme0n1.out‚47101.6455078125 disk.sda1.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.sda.out‚ 769.9921875 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 103.59375* proc.memory.percent‚11.795632044474285 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 103.59375" disk.dm-0.out‚47101.6455078125) proc.memory.availableMB‚ 1959.54296875 disk.drbd1356.in‚0 disk.drbd1550.out‚0óÆ*žÆ)‚(:“ Ž‚‹ÓÑÉݳ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚278.246' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.74866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-®'ELÇ)‚(:Á  Ž‚‹ÓÊÌÿ³ disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 90.0791015625 disk.drbd1226.out‚0 cpu‚25.335013529202957 network.sent‚ 1.6976532e+07 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1550.out‚0 network.recv‚ 1.240378e+06 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0) disk./.usagePercent‚70.51391929569431 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 770.2890625 disk.drbd2078.in‚0% disk./.usageGB‚1260.8604927062988 disk.drbd1673.in‚0 disk.drbd1938.in‚0# memory_percent‚92.0275847117106 disk.dm-0.in‚ 103.59375* proc.memory.percent‚11.795632044474285 disk.drbd1552.out‚0" disk.dm-0.out‚47102.1611328125 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd2111.in‚0 disk.sda2.out‚ 770.2890625 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 1959.30078125 proc.cpu.threads‚17 disk.drbd1828.in‚0% disk.nvme0n1.out‚47102.1611328125/ÛŸÈ)‚(:” ‚‹Óˆé¶³ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚309.996' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.33200000000001 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04å ¢MÉ)‚(:  ‚‹ÓÞÀÔ³) disk./.usagePercent‚70.51391545561316 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚47102.3486328125 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1959.09765625 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1510.out‚0 network.sent‚ 1.6979531e+07 disk.drbd1584.out‚0 disk.drbd1673.out‚0" disk.dm-0.out‚47102.3486328125 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 network.recv‚ 1.241369e+06 disk.sda2.out‚ 770.75390625 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.795632044474285 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0$ disk./.usageGB‚1260.860424041748 disk.dm-0.in‚ 103.59375 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.rbd1.out‚ 90.1416015625 disk.drbd1552.in‚0 disk.sda.out‚ 770.75390625 disk.drbd1931.in‚0 disk.drbd2111.in‚0# memory_percent‚92.0284112294515 cpu‚25.334956483104904 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 ­8·žÊ)‚(:“ ¬‚‹Ó¶ŠÏ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.128' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.04266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'å÷NË)‚(:à  ¬‚‹ÓÆÆî³ disk.drbd1285.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 103.59375# memory_percent‚92.0284112294515# proc.memory.rssMB‚ 2898.89453125 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 network.recv‚ 1.242511e+06 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 103.59375# disk.dm-0.out‚47103.10986328125 disk.drbd1285.out‚0 network.sent‚ 1.6983071e+07 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 90.3212890625 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 771.13671875 disk.drbd1817.out‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1959.09765625 cpu‚25.33488376989254 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0$ disk./.usageGB‚1260.860424041748& disk.nvme0n1.out‚47103.10986328125 disk.drbd1931.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51391545561316 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 771.13671875 disk.drbd1965.in‚0* proc.memory.percent‚11.795632044474285 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0w*ü½4Ì)/ _wandbruntime‚17643  _runtime‚17643@‰¼ÊÍ)Ä _step‚806 ' lr-AdamW/pg1‚2.6178493046535202e-05 ' lr-AdamW/pg2‚2.6178493046535202e-05 trainer/global_step‚9699 % _timestamp‚1.7848568803192322e+09 _runtime‚17643.676479525Ê4³VÏÎ)É ' lr-AdamW/pg1‚2.6178493046535202e-05 ' lr-AdamW/pg2‚2.6178493046535202e-05 trainer/global_step‚9699 % _timestamp‚1.7848568803192322e+09 _runtime‚17643.676479525 _step‚806¦ƒ8Î84Ï)/ _wandbruntime‚17643  _runtime‚17643gxš‘(Ð)ŠP LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2801574309838470 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55396922 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2792571184416768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10122589997568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10122589997568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12379051941888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174264875 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55396562 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2792571184416768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25847036 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84204996 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2802692428978176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2802692428978176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2802693800587776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84205002 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2792571210590208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 trainer/global_step‚9699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84204996 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90059879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29549886 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90069943 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12379107403264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2801575679877724 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764144928306176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2791470012489008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚17643.680922652 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0t^tQ Ñ)P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2802692428978176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2802692428978176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2802693800587776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84205002 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2792571210590208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 trainer/global_step‚9699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84204996 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90059879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29549886 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90069943 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12379107403264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2801575679877724 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764144928306176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2791470012489008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚17643.680922652 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2801574309838470 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55396922 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2792571184416768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10122589997568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10122589997568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12379051941888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174264875 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55396562 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2792571184416768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25847036 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84204996 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623§ +4Ò)/ _wandbruntime‚17645  _runtime‚17645™±æÙ'Ó)’N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2792850119880704 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10106662762500 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚17645.15135115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90068961 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764223882763264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174292408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90068961 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174282324 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84213363 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚776603818118144 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2801855502453252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90079039 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55402133 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25849676 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2792850093707264 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 trainer/global_step‚9699 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _step‚808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12380356803072 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10122267069440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10122267069440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2801854132413886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174292408 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2791748814262784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181·Dò'Ô)—N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _step‚808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12380356803072 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10122267069440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10122267069440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2801854132413886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174292408 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2791748814262784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2792850119880704 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10106662762500 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 _runtime‚17645.15135115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90068961 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764223882763264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174292408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90068961 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174282324 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84213363 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚776603818118144 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2801855502453252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90079039 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55402133 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25849676 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2792850093707264 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 trainer/global_step‚9699¨1û*š4Õ)/ _wandbruntime‚17645  _runtime‚17645Æ_ì Ö)„ 3loss/SRqqt1/assignment_loss‚0.49402144934277464 2loss/SRqqt2/assignment_loss‚0.5112727735016773 1loss/SRqqt2/detection_loss‚0.5952914586481678 /loss/FBt2/detection_loss‚0.4379076083130293 0loss/FRt1/assignment_loss‚1.6438694849453173 0loss/FRt2/assignment_loss‚0.7439562782431334 1loss/FBt1/assignment_loss‚0.11289329456111573 /loss/FBt1/detection_loss‚0.3454333963907427 epoch‚34 _runtime‚17645.188348284 1loss/FBt2/assignment_loss‚0.05628975240159295 /loss/FRt1/detection_loss‚0.5343111719967744 /loss/FRt2/detection_loss‚1.4645123394015669 trainer/global_step‚9699 % _timestamp‚1.7848568818311303e+09 1loss/SRqqt1/detection_loss‚0.9900596424004298 %loss/total_loss‚7.929818650146323 _step‚809Á8×)‰ _step‚809 3loss/SRqqt1/assignment_loss‚0.49402144934277464 2loss/SRqqt2/assignment_loss‚0.5112727735016773 1loss/SRqqt2/detection_loss‚0.5952914586481678 /loss/FBt2/detection_loss‚0.4379076083130293 0loss/FRt1/assignment_loss‚1.6438694849453173 0loss/FRt2/assignment_loss‚0.7439562782431334 1loss/FBt1/assignment_loss‚0.11289329456111573 /loss/FBt1/detection_loss‚0.3454333963907427 epoch‚34 _runtime‚17645.188348284 1loss/FBt2/assignment_loss‚0.05628975240159295 /loss/FRt1/detection_loss‚0.5343111719967744 /loss/FRt2/detection_loss‚1.4645123394015669 trainer/global_step‚9699 % _timestamp‚1.7848568818311303e+09 1loss/SRqqt1/detection_loss‚0.9900596424004298 %loss/total_loss‚7.929818650146323©¬w¶OØ)‚(:Ä  »‚‹Óö¸ï³ disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0# disk.dm-0.out‚47104.08544921875 disk.drbd2078.out‚0 disk.rbd1.out‚ 90.3837890625 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0% disk./.usageGB‚1260.8561515808105 disk.sda2.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚47104.08544921875 disk.drbd1817.out‚0 network.sent‚ 1.698631e+07 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd1550.out‚0$ memory_percent‚92.02853838602701) proc.memory.availableMB‚ 1959.06640625 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 103.59375* proc.memory.percent‚11.795632044474285 cpu‚25.33484004768229) disk./.usagePercent‚70.51367651723142 disk.sda.out‚ 771.52734375 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 771.52734375 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd2118.in‚0 disk.drbd1356.in‚0 network.recv‚ 1.243646e+06 disk.drbd2111.in‚0 disk.drbd1226.in‚0ÓŠV“Ù)‚(:ˆ »‚‹Ó¶‘ѳ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.923' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.641 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ wÚ^Ú)‚(:à  Ê‚‹ÓáÊõ³ disk.drbd1356.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1828.out‚0 disk.drbd1550.out‚0 cpu‚25.3347821452609 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚47104.79931640625 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2898.89453125) disk./.usagePercent‚70.51367651723142 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 771.87109375 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 103.59375 disk.rbd1.out‚ 90.3837890625 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1959.21484375 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0# memory_percent‚92.0279343922933I½3Âð network.recv‚ 1.246383e+06 disk.drbd1673.in‚0% disk./.usageGB‚1260.8561515808105 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.sda.out‚ 771.87109375 disk.drbd2041.out‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚47104.79931640625 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0* proc.memory.percent‚11.795632044474285 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 103.59375 network.sent‚ 1.7045733e+07åþˆEŸÛ)‚(:” Ê‚‹Ó̑׳ gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚308.676' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.89200000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O…ú žÜ)‚(:“ Ù‚‹Ó󡪳 gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚284.534' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.84466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑQÿüLÝ)‚(:Á  Ù‚‹Ó™—̳ disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.rbd1.out‚ 90.478515625 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.795632044474285 network.recv‚ 1.247424e+06 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0( proc.memory.availableMB‚ 1958.9140625% disk./.usageGB‚1260.8561515808105 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.02915827433267) disk./.usagePercent‚70.51367651723142 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2898.89453125 disk.sda2.out‚ 772.49609375 disk.drbd1289.in‚0% disk.nvme0n1.out‚47105.3662109375 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 cpu‚25.33472543737485 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 772.49609375 disk.nvme0n1.in‚ 103.59375 disk.drbd1552.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 103.59375 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1673.in‚0 network.sent‚ 1.7048726e+07 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0" disk.dm-0.out‚47105.3662109375jœ|ÄžÞ)‚(:“ è‚‹ÓÀÒ³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚249.541' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.18033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ¬" Cß)‚(:¸  è‚‹ÓÚíܳ disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 772.90625 disk.drbd2078.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0& proc.memory.availableMB‚ 1958.90625 disk.drbd1289.out‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚47105.9873046875 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 90.478515625# proc.memory.rssMB‚ 2898.89453125 disk.drbd1285.out‚0 disk.drbd1226.out‚0 network.sent‚ 1.7052412e+07 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.795632044474285% disk./.usageGB‚1260.8561515808105 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.sda2.out‚ 772.90625$ memory_percent‚92.02919006347656 network.recv‚ 1.248655e+06) disk./.usagePercent‚70.51367651723142" disk.dm-0.out‚47105.9873046875 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 103.59375 cpu‚25.3346956712801 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0mšçÛà)‚(:’ ÷‚‹Ó¬ë½³ gpu.0.gpu‚22.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚107.94' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.980000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U4jvFá)‚(:»  ÷‚‹Ó¦ôܳ disk.drbd2041.out‚0& proc.memory.availableMB‚ 1958.90625 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2898.89453125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.sda.in‚ 0.03515625$ memory_percent‚92.02919006347656 disk.drbd1289.out‚0" disk.dm-0.out‚47106.2060546875 disk.rbd1.out‚ 90.728515625 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.sda.out‚ 773.171875 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51367758392063 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1510.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 network.sent‚ 1.7054871e+07 disk.drbd1356.in‚0 disk.drbd1817.in‚0 cpu‚25.33465226999431 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.795632044474285 disk.sda2.out‚ 773.171875 disk.dm-0.in‚ 103.59375 disk.drbd1356.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 network.recv‚ 1.249584e+06% disk./.usageGB‚1260.8561706542969 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625% disk.nvme0n1.out‚47106.2060546875 disk.drbd1226.in‚0 disk.drbd2118.in‚0’:ãžâ)‚(:“ †ƒ‹Ó„êÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚291.464' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.15466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³9´:ã)‚(:¯  †ƒ‹ÓŸ½â³ disk.drbd1356.in‚0# proc.memory.rssMB‚ 2898.89453125 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 773.4765625 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.02919006347656 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 773.4765625 disk.dm-0.out‚ 47106.625 disk.nvme0n1.out‚ 47106.625 network.recv‚ 1.25049e+06 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51367651723142 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0* proc.memory.percent‚11.795632044474285% disk./.usageGB‚1260.8561515808105 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.rbd1.out‚ 90.908203125 proc.cpu.threads‚17 network.sent‚ 1.7057859e+07 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0& proc.memory.availableMB‚ 1958.90625 cpu‚25.334566573447052 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.dm-0.in‚ 103.59375 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1226.out‚0aQh#4ä)/ _wandbruntime‚17730  _runtime‚17730ý‹Ô‘Êå)Ä % _timestamp‚1.7848569671069882e+09 _runtime‚17730.464812798 _step‚810 ' lr-AdamW/pg1‚2.5621976624351286e-05 ' lr-AdamW/pg2‚2.5621976624351286e-05 trainer/global_step‚9749-û—kÏæ)É _runtime‚17730.464812798 _step‚810 ' lr-AdamW/pg1‚2.5621976624351286e-05 ' lr-AdamW/pg2‚2.5621976624351286e-05 trainer/global_step‚9749 % _timestamp‚1.7848569671069882e+09ª•CZ¦4ç)/ _wandbruntime‚17730  _runtime‚177309Û(è)ŠP PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90515831 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84623352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175139177 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2805410101177808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2816690454668800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _runtime‚17730.465229837 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10172434117632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29696270 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2806516648941568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84623352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚768074246603776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2806516675115008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2816689083059200 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10173779553792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84623346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55675652 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780515967970816 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2805410126605776 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2816690454668800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780516387329024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12441772634112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10173779553792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10172434117632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84623346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2816689083059200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2815565502410078 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848569671077383e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2806516675115008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12441721367040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2815566872449332 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90515831m(é)P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2805410101177808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2816690454668800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _runtime‚17730.465229837 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10172434117632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29696270 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2806516648941568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84623352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚768074246603776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2806516675115008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2816689083059200 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10173779553792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84623346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55675652 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780515967970816 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2805410126605776 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2816690454668800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780516387329024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12441772634112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10173779553792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10172434117632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84623346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2816689083059200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2815565502410078 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 % _timestamp‚1.7848569671077383e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2806516675115008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12441721367040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2815566872449332 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90515831 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90515831 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84623352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175139177«Bâý4ê)/ _wandbruntime‚17731  _runtime‚17731Ô  ë)”N REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84631719 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90534991 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2816970386474496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10157766645308 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25981671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29699513 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2805688928379552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90524913 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25981676 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚780595806852608 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2815845324985494 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84631713 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55680837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2805688902951584 trainer/global_step‚9749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55681189 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚768153201060864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚768152832969728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10156422033910 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84631719 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10173456625664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2816969014857728 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2806795558232064 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10173456625664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175166710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÿ¤ 'ì)™N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90534991 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚780595806852608 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2815845324985494 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84631713 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55680837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚347 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2805688902951584 trainer/global_step‚9749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55681189 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚768153201060864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚768152832969728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10156422033910 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84631719 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10173456625664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2816969014857728 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2806795558232064 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10173456625664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175166710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84631719 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90534991 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2816970386474496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10157766645308 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25981671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29699513 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2805688928379552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90524913 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25981676 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 Bƒ þ)‰P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10223623673856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10207849726364 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26111036 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2819350215294576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.o\/ΨŒn_train_batch_start/active.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚772003564901376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55954478 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10224969110016 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2830685737140224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2820462139639808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90981847 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29843111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚815 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85041696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55954142 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784507965182976 trainer/global_step‚9799 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2829556694981686 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2830685737140224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176023549 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2829558065020940 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2819350189866608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90971783 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29843442 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12504400281600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2820462113466368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85041696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚772003932992512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.784857053764499e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 _runtime‚17817.124927497 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90981847 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10224969110016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26111031 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10206505115078 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90971783 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2830687108749824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2820462139639808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2820462113466368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112XöóT(ÿ)ŽP `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12504400281600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2820462113466368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85041696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚772003932992512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.784857053764499e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 _runtime‚17817.124927497 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90981847 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10224969110016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26111031 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10206505115078 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90971783 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2830687108749824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2820462139639808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2820462113466368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10223623673856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10207849726364 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26111036 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2819350215294576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚772003564901376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55954478 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10224969110016 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2830685737140224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2820462139639808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90981847 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29843111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _step‚815 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85041696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55954142 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784507965182976 trainer/global_step‚9799 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2829556694981686 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2830685737140224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176023549 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2829558065020940 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2819350189866608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90971783 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29843442¯ xÜ4€*/ _wandbruntime‚17818  _runtime‚17818ÅÝS~'*”N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚772082151267328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55959703 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85050063 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85050069 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10225991625216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10224646181888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784588224720384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚17818.600296292 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2820741022756864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2830965668938752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12505705361920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2829837887596468 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176030928 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2830967040555520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90980865 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12505660393472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _step‚816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2820741048930304 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176041012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55960037 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2819628991640384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85050063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29846361 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176030928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10207525916718 % _timestamp‚1.7848570552426581e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2820741048930304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2830967040555520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚784587811660800 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚772082519358464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10208870528116 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90990943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2819629017068352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10225991625216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10224646181888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85050069 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90990943 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55959703 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85050063 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85050069 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10225991625216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10224646181888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784588224720384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚17818.600296292 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2820741022756864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2830965668938752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12505705361920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2829837887596468 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176030928 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2830967040555520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90980865 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12505660393472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 _step‚816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 413059584 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2820741048930304 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176041012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55960037 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2819628991640384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85050063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29846361 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44968448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176030928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10207525916718 % _timestamp‚1.7848570552426581e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2820741048930304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2830967040555520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚784587811660800 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚772082519358464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10208870528116 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90990943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2819629017068352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10225991625216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10224646181888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85050069 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90990943 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 F&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7159651517868042 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8"Purity/1FRt1SRqqt0FBt/event_purity‚0.042228814214468 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028323128819465637 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2663504481315613 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25482454895973206 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87980717420578 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8339646458625793 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025991441681981087 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7176882028579712 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6273136734962463 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 +particle/specificity‚0.9121870398521423 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8732222318649292 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6600449681282043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2486695796251297 particle/accuracy_1_of_1‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.687153160572052 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6431139707565308 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6355268359184265 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3321465849876404 6validation_average_jet_accuracy‚0.3960913121700287 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22203834354877472 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 particle/accuracy_1_of_3‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6426337361335754 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5984376072883606 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 *jet/accuracy_3_of_4‚0.1148163229227066 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6543744206428528 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7744976282119751 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 % _timestamp‚1.7848570746869578e+09 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5095995664596558 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.729330837726593 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 *jet/accuracy_1_of_4‚0.9745305180549622 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6895580887794495 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8786140084266663 +jet/accuracy_2_of_2‚0.16996023058891296 particle/accuracy_2_of_2‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14020155370235443 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9512740969657898 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27605530619621277 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.34501025080680847 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.764332115650177 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6387454867362976 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34041881561279297 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.15538455545902252 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46657970547676086 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2883565425872803 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3321465849876404 *jet/accuracy_1_of_2‚0.7739240527153015 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚34 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3742051422595978 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6524098515510559 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05112128332257271 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15333005785942078 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3234763741493225 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 +jet/accuracy_3_of_3‚0.07680032402276993 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38422349095344543 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8709415197372437 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7693763375282288 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7057548761367798 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.6042532920837402 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7003585696220398 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8381279110908508 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8703102469444275 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3790886700153351 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4548988938331604 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8555194139480591 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.608826756477356 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.723979651927948 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6373162269592285 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9242881536483765 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7105292677879333 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06063298135995865 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25210967659950256 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7314446568489075 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6842209696769714 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.690421998500824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 )loss/val_total_loss‚7.791593460424555 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 trainer/global_step‚9799 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7694655060768127 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1217380240559578 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 *jet/accuracy_1_of_1‚0.3613065183162689 /particle/accuracy_4_of_4‚0.9685583114624023 _step‚818 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27307558059692383 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 particle/accuracy_3_of_4‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8752780556678772 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3442489802837372 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07829215377569199 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6614765524864197 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6730038523674011 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28637972474098206 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8786140084266663 *jet/accuracy_1_of_3‚0.9326351284980774 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27588602900505066 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _runtime‚17838.045031198 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7172861695289612 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7976862788200378 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5196612477302551 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2625666856765747 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8026006817817688 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8033213019371033¬µØ3Œ*v & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8703102469444275 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3790886700153351 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4548988938331604 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8555194139480591 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.608826756477356 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.723979651927948 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6373162269592285 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9242881536483765 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7105292677879333 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06063298135995865 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.25210967659950256 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7314446568489075 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6842209696769714 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.690421998500824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 )loss/val_total_loss‚7.791593460424555 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 trainer/global_step‚9799 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7694655060768127 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1217380240559578 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 *jet/accuracy_1_of_1‚0.3613065183162689 /particle/accuracy_4_of_4‚0.9685583114624023 _step‚818 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27307558059692383 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 particle/accuracy_3_of_4‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8752780556678772 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3442489802837372 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07829215377569199 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6614765524864197 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6730038523674011 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28637972474098206 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8786140084266663 *jet/accuracy_1_of_3‚0.9326351284980774 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27588602900505066 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _runtime‚17838.045031198 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7172861695289612 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7976862788200378 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5196612477302551 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2625666856765747 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8026006817817688 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8033213019371033 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014799253083765507 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8744279146194458 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3818655014038086 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6337786316871643 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014799253083765507 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 (particle/accuracy‚0.8543527722358704 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.33207330107688904 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023122143000364304 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1707613617181778 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6397873759269714 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8032537698745728 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3602781891822815 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.330116331577301 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.550432026386261 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 +jet/accuracy_4_of_4‚0.02295798622071743 /particle/accuracy_3_of_3‚0.9993678331375122 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15772055089473724 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22553355991840363 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6484570503234863 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7605382204055786 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6629474759101868 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34639933705329895 8 P¤_ëãurity/*FRt2SRqqt0FBt/FRt_purity‚0.02145417034626007 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6387454867362976 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5837894082069397 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27029797434806824 particle/accuracy_2_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7159651517868042 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 8"Purity/1FRt1SRqqt0FBt/event_purity‚0.042228814214468 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028323128819465637 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2663504481315613 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25482454895973206 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87980717420578 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8339646458625793 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025991441681981087 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7176882028579712 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6273136734962463 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 +particle/specificity‚0.9121870398521423 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8732222318649292 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6600449681282043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2486695796251297 particle/accuracy_1_of_1‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.687153160572052 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6431139707565308 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6355268359184265 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3321465849876404 6validation_average_jet_accuracy‚0.3960913121700287 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22203834354877472 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 particle/accuracy_1_of_3‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6426337361335754 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5984376072883606 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 *jet/accuracy_3_of_4‚0.1148163229227066 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6543744206428528 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7744976282119751 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 % _timestamp‚1.7848570746869578e+09 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5095995664596558 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.729330837726593 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 *jet/accuracy_1_of_4‚0.9745305180549622 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6895580887794495 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8786140084266663 +jet/accuracy_2_of_2‚0.16996023058891296 particle/accuracy_2_of_2‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14020155370235443 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9512740969657898 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27605530619621277 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.34501025080680847 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.764332115650177 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6387454867362976 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34041881561279297 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.15538455545902252 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46657970547676086 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2883565425872803 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3321465849876404 *jet/accuracy_1_of_2‚0.7739240527153015 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚34 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3742051422595978 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6524098515510559 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05112128332257271 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15333005785942078 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3234763741493225 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 +jet/accuracy_3_of_3‚0.07680032402276993 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38422349095344543 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8709415197372437 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7693763375282288 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7057548761367798 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.6042532920837402 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7003585696220398 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8381279110908508 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632²Ù”wßž*‚(:“ þƒ‹Ó¥Ðȳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚260.245' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.74833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zJŽ*‚(:¿  þƒ‹Ó¥éê³ disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 network.recv‚ 1.261546e+06 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0" proc.memory.rssMB‚ 2910.7421875 disk.drbd1931.in‚0$ memory_percent‚92.07626978556314 network.sent‚ 1.7195946e+07 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.sda.out‚ 776.21484375 disk.drbd1931.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd1226.in‚0* proc.memory.percent‚11.843840281168621 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0( proc.memory.availableMB‚ 1947.3359375# disk./.usageGB‚1260.85640335083 disk.drbd1584.in‚0 disk.rbd1.out‚ 91.30078125& disk.nvme0n1.out‚47109.31787109375 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd2041.in‚0 cpu‚25.336836724140856 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 776.21484375 disk.drbd1828.out‚0 disk.drbd1550.out‚0( disk./.usagePercent‚70.5136905975289 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0# disk.dm-0.out‚47109.31787109375 disk.drbd2078.in‚0þU(I*‚(:¾  „‹ÓˆÒ€´ disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 network.recv‚ 1.264237e+06( disk./.usagePercent‚70.5136905975289 disk.drbd1289.out‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.843840281168621 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 103.59375 cpu‚25.336791392628474 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 91.36328125& disk.nvme0n1.out‚47109.50146484375 disk.drbd1289.in‚0 disk.drbd1584.out‚0$ memory_percent‚92.07719167073569 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2910.7421875 disk.drbd1673.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚47109.50146484375 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0' proc.memory.availableMB‚ 1947.109375 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.sda1.in‚0# disk./.usageGB‚1260.85640335083 disk.drbd1550.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 776.62890625 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 network.sent‚ 1.7249851e+07 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 776.62890625 ¼Þž*‚(:“ „‹Óݨ޳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.296' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.09866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ ï“‘*‚(:ˆ œ„‹ÓÁßdz gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.569' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡J»ÇC’*‚(:¸  œ„‹Ó–ÿè³' proc.memory.availableMB‚ 1947.109375 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚47109.87353515625 disk.drbd1285.in‚0 disk.sda.out‚ 776.96875 disk.drbd2078.out‚0 disk.drbd1356.in‚0 network.recv‚ 1.265153e+06 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 cpu‚25.336733224107444 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.843840281168621 network.sent‚ 1.7252893e+07 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 103.59375& disk.nvme0n1.out‚47109.87353515625 disk.drbd2078.in‚0( disk./.usagePercent‚70.5136905975289 disk.nvme0n1.in‚ 103.59375 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0$ memory_percent‚92.07719167073569 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0" proc.memory.rssMB‚ 2910.7421875 disk.drbd1817.out‚0 disk.sda2.out‚ 776.96875 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0# disk./.usageGB‚1260.85640335083 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.rbd1.out‚ 91.36328125 disk.drbd1584.out‚0 disk.drbd1931.in‚0¯BšŸ“*‚(:” «„‹Ó£˜â³ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.318' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.10599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïâc¼F”*‚(:»  «„‹Óõ°…´ disk.sda.out‚ 777.296875 disk.drbd1550.out‚0) disk./.usagePercent‚70.51371790477255 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 network.recv‚ 1.266211e+06 disk.drbd2111.out‚0 disk.dm-0.in‚ 103.59375 disk.rbd1.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.843840281168621 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.rbd1.out‚ 91.4970703125 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚47110.05712890625 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 103.59375 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0% proc.memory.availableMB‚ 1947.0625 cpu‚25.336673937566793$ memory_percent‚92.07738240559897" proc.memory.rssMB‚ 2910.7421875# disk./.usageGB‚1260.85689163208 disk.drbd1584.out‚0 disk.drbd2118.out‚0# disk.dm-0.out‚47110.05712890625 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 777.296875 disk.drbd1550.in‚0 network.sent‚ 1.7255991e+07 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0¼åÔež•*‚(:“ º„‹Óö®Ô³ gpu.0.gpu‚75.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.785' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.26166666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~f{D–*‚(:¹  º„‹Ó”÷û³ disk.drbd2041.out‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.843840281168621 disk.drbd2118.out‚0 disk.sda.out‚ 777.65625 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.dm-0.out‚47110.43359375 disk.drbd1965.in‚0 disk.rbd1.out‚ 91.4970703125 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 network.sent‚ 1.7258972e+07 disk.drbd2078.out‚0 disk.drbd1356.out‚0$ memory_percent‚92.07812945048013 disk.drbd1356.in‚0# disk.nvme0n1.out‚47110.43359375 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 cpu‚25.336601707455202) disk./.usagePercent‚70.51373241174572 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 103.59375" proc.memory.rssMB‚ 2910.7421875 disk.drbd2078.in‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 103.59375 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 network.recv‚ 1.267128e+06 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 777.65625 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1550.out‚0% disk./.usageGB‚1260.8571510314941 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1946.87890625¨ð}“4—*/ _wandbruntime‚17923  _runtime‚17923§vsǘ*Á & lr-AdamW/pg1‚2.452197019526526e-05 & lr-AdamW/pg2‚2.452197019526526e-05 trainer/global_step‚9849 $ _timestamp‚1.784857159921793e+09 _runtime‚17923.279054194 _step‚819shÌ™*Æ _runtime‚17923.279054194 _step‚819 & lr-AdamW/pg1‚2.452197019526526e-05 & lr-AdamW/pg2‚2.452197019526526e-05 trainer/global_step‚9849 $ _timestamp‚1.784857159921793e+09³(B 4š*/ _wandbruntime‚17923  _runtime‚17923Xé½Ñ(›*ˆP PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85549484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2836968280889344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚775982250803200 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12574656492544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30020321 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12574605225472 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2847248686382592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 $ _timestamp‚1.784857159922882e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10281777102848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚788556907295744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10263217837195 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _runtime‚17923.28293489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26249923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30019975 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2847250057992192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177024520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56269898 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91475042 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10280431666688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91475042 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2847248686382592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 $ _timestamp‚1.784857159922882e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10281777102848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚788556907295744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10263217837195 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 _runtime‚17923.28293489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26249923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30019975 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2847250057992192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177024520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KADeviceStatsMonitor.on_train_batch_sF—òâtart/active.small_pool.current‚10064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56269898 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91475042 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10280431666688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91475042 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30020321 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12574605225472 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10282799618048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2836128106908320 trainer/global_step‚9849 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10265583250233 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177052053 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85557845 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91484124 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177041969 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2837247164006400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚776060469078016 _step‚821 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10281454174720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91494202 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56275494 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2847529989797888 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7848571613979545e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2837247190179840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177041969 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚776060837169152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12575855539712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2836128081480352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2837247190179840 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10281454174720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85557845 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10282799618048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85557851 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91494202 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56275850 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚788636750260224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚17924.755556369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30023282 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10264238638835 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91484124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248iG[ú'Ÿ*™N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85557845 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91484124 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177041969 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2837247164006400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚776060469078016 _step‚821 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10281454174720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91494202 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56275494 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2847529989797888 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7848571613979545e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2837247190179840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177041969 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚776060837169152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12575855539712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2836128081480352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2837247190179840 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10281454174720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85557845 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10282799618048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85557851 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91494202 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56275850 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚788636750260224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚17924.755556369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30023282 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10264238638835 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91484124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85557851 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2847528618181120 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10282799618048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2836128106908320 trainer/global_step‚9849 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10265583250233 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177052053 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181µd1ÉA4 */ _wandbruntime‚17924  _runtime‚17924¿OÕ— ¡*„ &loss/total_loss‚7.4977533191143095 1loss/SRqqt2/assignment_loss‚0.442947360848998 2loss/FBt1/assignment_loss‚0.004474452466744361 2loss/FBt2/assignment_loss‚0.018690032661997813 /loss/FRt2/detection_loss‚1.3354061683713798 /loss/FRt1/detection_loss‚0.5133464266650967 trainer/global_step‚9849 _runtime‚17924.791590494 _step‚822 /loss/FBt1/detection_loss‚0.4549454766205482 /loss/FRt1/assignment_loss‚1.554375447249767 2loss/SRqqt1/assignment_loss‚0.5418701897055859 1loss/SRqqt1/detection_loss‚0.7757117921921624 1loss/SRqqt2/detection_loss‚0.5951414570816279 0loss/FRt2/assignment_loss‚0.6540192529187451 /loss/FBt2/detection_loss‚0.6068252623316569 epoch‚35 % _timestamp‚1.7848571614344215e+095ý¢*‰ 0loss/FRt2/assignment_loss‚0.6540192529187451 /loss/FBt2/detection_loss‚0.6068252623316569 epoch‚35 % _timestamp‚1.7848571614344215e+09 &loss/total_loss‚7.4977533191143095 1loss/SRqqt2/assignment_loss‚0.442947360848998 2loss/FBt1/assignment_loss‚0.004474452466744361 2loss/FBt2/assignment_loss‚0.018690032661997813 /loss/FRt2/detection_loss‚1.3354061683713798 /loss/FRt1/detection_loss‚0.5133464266650967 trainer/global_step‚9849 _runtime‚17924.791590494 _step‚822 /loss/FBt1/detection_loss‚0.4549454766205482 /loss/FRt1/assignment_loss‚1.554375447249767 2loss/SRqqt1/assignment_loss‚0.5418701897055859 1loss/SRqqt1/detection_loss‚0.7757117921921624 1loss/SRqqt2/detection_loss‚0.5951414570816279¶1†Rž£*‚(:“ É„‹Óă¶³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.415' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.47166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ã9I¤*‚(:¾  É„‹Ó¸­Õ³$ memory_percent‚92.08130836486816% disk./.usageGB‚1260.8572425842285 disk.drbd1584.out‚0 network.recv‚ 1.268014e+06 disk.drbd2111.in‚0 disk.sda2.out‚ 778.12109375 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2910.7421875 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 network.sent‚ 1.7261965e+07 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 103.59375 proc.cpu.threads‚17* proc.memory.percent‚11.843840281168621 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2111.out‚0 disk.rbd1.out‚ 91.5595703125 disk.drbd1965.out‚0 cpu‚25.336544173710912 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 778.12109375 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0( disk./.usagePercent‚70.5137375318539 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1946.09765625 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0# disk.nvme0n1.out‚47110.63671875 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 103.59375 disk.dm-0.out‚47110.63671875{Îg¥*‚(:’ Ø„‹ÓåÁÚ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚272.31' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.77000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã RtO¦*‚(:Ä  Ø„‹ÓºÝø³ disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 network.recv‚ 1.270765e+06 disk.drbd1510.in‚0 disk.drbd1510.out‚0 cpu‚25.336485054633783 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 778.44140625 disk.drbd1289.out‚0" proc.memory.rssMB‚ 2910.7421875 disk.sda1.in‚0 disk.sda.in‚ 0.03515625# disk.dm-0.out‚47111.17919921875 disk.drbd1965.in‚0 disk.drbd1673.in‚0& disk.nvme0n1.out‚47111.17919921875) proc.memory.availableMB‚ 1946.22265625 network.sent‚ 1.7322642e+07 disk.dm-0.in‚ 103.59375 disk.drbd1356.in‚0 disk.drbd2118.in‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0% disk./.usageGB‚1260.8572425842285 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.843840281168621 disk.sda.out‚ 778.44140625 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0( disk./.usagePercent‚70.5137375318539 disk.rbd1.out‚ 91.5595703125$ memory_percent‚92.08079973856607 disk.sda2.in‚ 0.03515625 disk.drbd1965.out‚0Š›ž=§*‚(:»  ç„‹Ó©äù³ disk.sda2.in‚ 0.03515625& proc.memory.availableMB‚ 1946.21875 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 778.71875 disk.dm-0.in‚ 103.59375 disk.drbd1965.out‚0 disk.drbd1817.out‚0 disk.drbd1931.6·^8 out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0# disk.dm-0.out‚47111.60888671875 disk.drbd2041.out‚0* proc.memory.percent‚11.843840281168621 proc.cpu.threads‚17 disk.drbd2078.out‚0 network.sent‚ 1.7325099e+07 disk.drbd1226.in‚0 disk.rbd1.out‚ 91.8017578125& disk.nvme0n1.out‚47111.60888671875 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0" proc.memory.rssMB‚ 2910.7421875 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 network.recv‚ 1.271651e+06) disk./.usagePercent‚70.51372899834027 disk.nvme0n1.in‚ 103.59375 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 cpu‚25.336440693739586 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 778.71875 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0$ disk./.usageGB‚1260.857089996338 disk.drbd1817.in‚0 disk.drbd2111.in‚0$ memory_percent‚92.08081563313803XS£kž¨*‚(:“ ç„‹ÓÃêس gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.572' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.85733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FªJ©*‚(:¿  ö„‹Óªþà³ disk.drbd1550.out‚0) proc.memory.availableMB‚ 1946.43359375 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 91.9814453125 disk.drbd1584.in‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚47112.48681640625 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51372899834027 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0" proc.memory.rssMB‚ 2910.7421875 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 cpu‚25.336355362972874 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.sda.out‚ 779.078125 disk.sda2.out‚ 779.078125 disk.drbd1817.out‚0 network.sent‚ 1.7328106e+07 disk.drbd1931.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚47112.48681640625 disk.drbd1285.in‚0 disk.drbd1510.out‚0 network.recv‚ 1.272623e+06# memory_percent‚92.0799414316813* proc.memory.percent‚11.843840281168621$ disk./.usageGB‚1260.857089996338 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 103.59375 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0m%Ÿª*‚(:” ö„‹ÓÀÓö³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚303.057' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.01900000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 p6(ž«*‚(:“ ……‹Óʇڳ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚284.417' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.80566666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01¡N¬*‚(:à  ……‹Óáüù³ disk.drbd2078.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 103.59375 network.recv‚ 1.273502e+06 disk.drbd1938.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 779.32421875 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 network.sent‚ 1.7331101e+07 disk.drbd1965.out‚0& disk.nvme0n1.out‚47112.72119140625 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 779.32421875 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.843840281168621 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51372899834027 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd2111.in‚0$ memory_percent‚92.08095868428549) proc.memory.availableMB‚ 1946.18359375 cpu‚25.33632451894217 disk.sda.in‚ 0.03515625 disk.drbd1828.out‚0 disk.rbd1.out‚ 92.0439453125 disk.nvme0n1.in‚ 103.59375$ disk./.usageGB‚1260.857089996338 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0" proc.memory.rssMB‚ 2910.7421875 proc.cpu.threads‚17 disk.drbd1289.in‚0# disk.dm-0.out‚47112.72119140625 disk.drbd2078.out‚0 disk.drbd2041.out‚0}Ôý…Ÿ­*‚(:” ”…‹ÓÍúÒ³ gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.477' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.49233333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z•N®*‚(:à  ”…‹Ó»Ïî³ disk.drbd2041.in‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.843840281168621 network.sent‚ 1.7334144e+07 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 779.79296875 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 103.59375 disk.drbd1226.in‚0 disk.drbd1552.out‚0( proc.memory.availableMB‚ 1946.1796875 cpu‚25.336266840207948 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚47113.49169921875 disk.drbd1289.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.out‚ 779.79296875 disk.drbd1285.out‚0 disk.dm-0.in‚ 103.59375 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0# disk.dm-0.out‚47113.49169921875) disk./.usagePercent‚70.51372899834027$ disk./.usageGB‚1260.857089996338 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 92.2236328125$ memory_percent‚92.08097457885742 network.recv‚ 1.274498e+06 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625" proc.memory.rssMB‚ 2910.7421875 disk.sda1.out‚0 disk.sda.in‚ 0.03515625@Õ •4¯*/ _wandbruntime‚18010  _runtime‚18010Iö­‘Ê°*Ä ' lr-AdamW/pg1‚2.3978624374003586e-05 ' lr-AdamW/pg2‚2.3978624374003586e-05 trainer/global_step‚9899 % _timestamp‚1.7848572466880164e+09 _runtime‚18010.045279212 _step‚823›2 þϱ*É ' lr-AdamW/pg1‚2.3978624374003586e-05 ' lr-AdamW/pg2‚2.3978624374003586e-05 trainer/global_step‚9899 % _timestamp‚1.7848572466880164e+09 _runtime‚18010.045279212 _step‚823·;f34²*/ _wandbruntime‚18010  _runtime‚18010xšç(³*‰P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26381928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2850913719240704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56549904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚779911201009664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚18010.046644367 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10331621222912 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2861246712073216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2860103690115379 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177908892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12637173613056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10315666331289 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91941058 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2850913745414144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91930994 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861245340463616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10314321720003 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2850913719240704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10332966659072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85967828 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30167976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2849789393823344 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2849789368395376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚779911569100800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177898822 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2860105060154633 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10331621222912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85967834 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85967834 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824S^‰Ê(´*ŽP M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56549542 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26381923 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 trainer/global_step‚9899 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85967828 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861245340463616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10314321720003 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2850913719240704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10332966659072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85967828 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30167976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚357 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2849789393823344 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2849789368395376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚779911569100800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177898822 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2860105060154633 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10331621222912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85967834 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85967834 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26381928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2850913719240704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56549904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚779911201009664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 _runtime‚18010.046644367 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10331621222912 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2861246712073216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2860103690115379 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177908892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12637173613056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10315666331289 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91941058 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2850913745414144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91930994 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30170897 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177926355 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12638369364992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26384568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177926355 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2851192654704640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91940076 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 477098526(ÛB3K72 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚9899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85976195 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177916271 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12638418527744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10332643730944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2850068170169152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177916271 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10316687133041 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2861526643878912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85976195 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91940076 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚779989787375616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91950154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2851192628531200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚792628573994496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7848572483501053e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10332643730944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26384563 _step‚825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2861525272262144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56555110 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30170897 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177926355 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12638369364992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26384568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177926355 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2851192654704640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91940076 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚9899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85976195 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177916271 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12638418527744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10332643730944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2850068170169152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177916271 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10316687133041 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2861526643878912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85976195 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91940076 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚779989787375616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91950154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2851192628531200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚792628573994496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7848572483501053e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10332643730944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26384563 _step‚825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2861525272262144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56555110 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178773124 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2875243366154240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚796540421374464 WJDeviceSÒ**ÇtatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚18096.892924777 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2874094882686987 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26513928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚377 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92397010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚828 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7848573335317452e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2864859183765504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2875241994544640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10384156215296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2863729482512144 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2864859209938944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86386184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚783840519307264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10365425602811 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92386946 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2875243366154240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚783840887398400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30316022 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12699472223232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2864859209938944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92397010 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56829950 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2875241994544640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10366770214097 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚103828107791360‰6(Ì*P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30316022 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12699472223232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2864859209938944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92397010 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56829950 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2875241994544640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10366770214097 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10382810779136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86386178 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2874096252726241 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86386178 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10382810779136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178783194 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12699533976064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92386946 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56829568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10384156215296 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2864859183765504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178773124 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2875243366154240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚796540421374464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚18096.892924777 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2874094882686987 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26513928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚377 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92397010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚828 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7848573335317452e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2864859183765504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2875241994544640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10384156215296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2863729482512144 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2864859209938944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86386184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚783840519307264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10365425602811 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92386946 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2875243366154240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚783840887398400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248¼q‹ü4Í*/ _wandbruntime‚18098  _runtime‚18098Háì'Î*”N QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2875521926343168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178800657 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2875523297959936 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86394551 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2865138093056000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2874374705262403 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178790573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26516568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚783919105673216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚18098.361649807 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2865138119229440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚377 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12700782937600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178790573 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚372 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚796619835253760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10383833287168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2874376075301769 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2864008284285920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86394545 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30318937 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10383833287168 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86394551 _step‚829 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10385178730496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178800657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2865138119229440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56835505 trainer/global_step‚9949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92406106 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86394545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚796620256701952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56835128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12700729580544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848573350040414e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92396028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30318565 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚783919473764352 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2875523297959936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10367791015849 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824ËI„³Ï*™N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178800657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2865138119229440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56835505 trainer/global_step‚9949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92406106 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86394545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚796620256701952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56835128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12700729580544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848573350040414e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92396028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30318565 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚783919473764352 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2875523297959936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10367791015849 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2875521926343168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178800657 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2875523297959936 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86394551 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2865138093056000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2874374705262403 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178790573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26516568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚783919105673216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚18098.361649807 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2865138119229440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚377 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12700782937600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178790573 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚372 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚796619835253760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10383833287168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2874376075301769 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2864008284285920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86394545 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30318937 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10383833287168 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86394551 _step‚829 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10385178730496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0½ ®Z4Ð*/ _wandbruntime‚18098  _runtime‚180985>Uð Ñ*† .loss/FRt1/detection_loss‚0.537353455487638 0loss/SRqqt1/detection_loss‚0.912313092748004 /loss/FBt1/detection_loss‚0.5185336256399751 % _timestamp‚1.7848573350432994e+09 0loss/FRt1/assignment_loss‚1.5716089634500312 3loss/SRqqt2/assignment_loss‚0.49443103869069027 3loss/FBt1/assignment_loss‚0.0037547482382898067 2loss/SRqqt1/assignment_loss‚0.5109779012783321 2loss/FBt2/assignment_loss‚0.004252036836682782 _runtime‚18098.400402463 /loss/FRt2/detection_loss‚1.4234585536289073 1loss/SRqqt2/detection_loss‚0.6005537998753523 0loss/FBt2/detection_loss‚0.44765574219668836 %loss/total_loss‚7.676922986794397 epoch‚35 trainer/global_step‚9949 _step‚830 0loss/FRt2/assignment_loss‚0.6520300287238054‹øzáÒ*‹ /loss/FRt2/detection_loss‚1.4234585536289073 1loss/SRqqt2/detection_loss‚0.6005537998753523 0loss/FBt2/detection_loss‚0.44765574219668836 %loss/total_loss‚7.676922986794397 epoch‚35 trainer/global_step‚9949 _step‚830 0loss/FRt2/assignment_loss‚0.6520300287238054 .loss/FRt1/detection_loss‚0.537353455487638 0loss/SRqqt1/detection_loss‚0.912313092748004 /loss/FBt1/detection_loss‚0.5185336256399751 % _timestamp‚1.7848573350432994e+09 0loss/FRt1/assignment_loss‚1.5716089634500312 3loss/SRqqt2/assignment_loss‚0.49443103869069027 3loss/FBt1/assignment_loss‚0.0037547482382898067 2loss/SRqqt1/assignment_loss‚0.5109779012783321 2loss/FBt2/assignment_loss‚0.004252036836682782 _runtime‚18098.400402463¾õê%“Ó*‚(:ˆ ý…‹ÓäúÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.157' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.719 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôî„MÔ*‚(:  ý…‹Ó¶Æÿ³ disk.drbd1965.out‚0& disk.nvme0n1.out‚47116.42626953125 disk.sda1.in‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17 network.sent‚ 1.7412985e+07 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0" proc.memory.rssMB‚ 2910.7421875 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 782.04296875 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0# disk.dm-0.out‚47116.42626953125 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0$ memory_percent‚92.08345413208008 cpu‚25.33587733973629 disk.dm-0.in‚ 103.59375 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51373198507004% disk./.usageGB‚1260.8571434020996 disk.rbd1.out‚ 92.716796875 network.recv‚ 1.283315e+06 disk.drbd2118.in‚0 disk.sda.out‚ 782.04296875 disk.drbd1673.out‚0( proc.memory.availableMB‚ 1945.5703125* proc.memory.percent‚11.843840281168621 disk.nvme0n1.in‚ 103.59375 disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625ƒlÈŸÕ*‚(:” Œ†‹Ó‰Çѳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚310.339' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.44633333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë'ÄKÖ*‚(:À  Œ†‹ÓÁ¯ò³ network.sent‚ 1.7472972e+07 disk.nvme0n1.in‚ 103.59375 disk.sda1.in‚0$ memory_percent‚92.08401044209799" proc.memory.rssMB‚ 2910.7421875 network.recv‚ 1.286581e+06 disk.sda2.out‚ 782.51953125$ disk.nvme0n1.out‚47116.736328125 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1938.in‚0! disk.dm-0.out‚47116.736328125 disk.drbd2078.out‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.8571243286133 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 103.59375 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51373091838083 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 92.716796875 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1945.43359375 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 cpu‚25.335834229143337 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1552.in‚0 disk.sda.out‚ 782.51953125 disk.drbd1226.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.843840281168621w0$ñž×*‚(:“ ›†‹Óަ³ gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.619' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.20633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\#DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚391 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _step‚832 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92852962 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57110136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2889238648625664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚787769837604864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10417874096905 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10434000335360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92842898 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86804528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30464604 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10416529485619 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12761710318080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26645928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832Ö¾>I ä*P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚396 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2877669545772976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10434000335360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92842898 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86804528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30464604 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10416529485619 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12761710318080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_ÀJyÌstart/inactive_split.large_pool.allocated‚26645928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚391 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _step‚832 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92852962 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57110136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2889238648625664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚787769837604864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10417874096905 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57115702 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10435022843392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92851980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26648563 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚800611799985152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚787848423970816 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179674959 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10435022843392 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2879083583754240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚397 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2888367267873377 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2877948372974720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57116099 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2877948347546752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10417550287259 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179674959 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2889519952040960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30467531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86812895 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2889518580424192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10418894898657 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92862058 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2879083557580800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26648568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30467139 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2889519952040960 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92851980 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2879083583754240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10436368286720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚800611368051200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚18185.168450778 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86812901 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 # _timestamp‚1.78485742181086e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179664875 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12763007923200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179664875¨–ƒ²'ç*—N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10436368286720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚800611368051200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚18185.168450778 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86812901 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 # _timestamp‚1.78485742181086e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179664875 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12763007923200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179664875 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57115702 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10435022843392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92851980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26648563 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚800611799985152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚787848423970816 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179674959 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10435022843392 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2879083583754240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚397 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2888367267873377 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2877948372974720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57116099 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2877948347546752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10417550287259 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179674959 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2889519952040960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30467531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86812895 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2889518580424192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10418894898657 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92862058 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2879083557580800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26648568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30467139 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2889519952040960 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92851980 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2879083583754240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5ÁV¦Á4è*/ _wandbruntime‚18185  _runtime‚18185ðµ~ é*‡ 3loss/SRqqt2/assignment_loss‚0.48199073606443277 3loss/FBt1/assignment_loss‚0.0002228457905480615 1loss/FBt2/assignment_loss‚0.02307086268028228 /loss/FRt1/detection_loss‚0.5063318148517753 /loss/FRt2/detection_loss‚1.4358602596917949 _runtime‚18185.207377094 1loss/SRqqt1/detection_loss‚0.8773320300049527 /loss/FBt2/detection_loss‚0.4184828165070541 0loss/FRt1/assignment_loss‚1.6378468453771022 1loss/SRqqt2/detection_loss‚0.5531399541300348 epoch‚35 trainer/global_step‚9999 % _timestamp‚1.7848574218502007e+09 _step‚834 /loss/FBt1/detection_loss‚0.5116146995375553 %loss/total_loss‚7.455335760039541 0loss/FRt2/assignment_loss‚0.5427330231943795 3loss/SRqqt1/assignment_loss‚0.46670987220962923–§¿ê*Œ /loss/FBt2/detection_loss‚0.4184828165070541 0loss/FRt1/assignment_loss‚1.6378468453771022 1loss/SRqqt2/detection_loss‚0.5531399541300348 epoch‚35 trainer/global_step‚9999 % _timestamp‚1.7848574218502007e+09 _step‚834 /loss/FBt1/detection_loss‚0.5116146995375553 %loss/total_loss‚7.455335760039541 0loss/FRt2/assignment_loss‚0.5427330231943795 3loss/SRqqt1/assignment_loss‚0.46670987220962923 3loss/SRqqt2/assignment_loss‚0.48199073606443277 3loss/FBt1/assignment_loss‚0.0002228457905480615 1loss/FBt2/assignment_loss‚0.02307086268028228 /loss/FRt1/detection_loss‚0.5063318148517753 /loss/FRt2/detection_loss‚1.4358602596917949 _runtime‚18185.207377094 1loss/SRqqt1/detection_loss‚0.8773320300049527Âýj˜àŸë*‚(:” ׆‹Ó‘§§³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚313.414' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.47133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÀä@Gì*‚(:¼  ׆‹Ó‡€Ê³ disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 cpu‚25.335591705244344 network.sent‚ 1.7491544e+07) disk./.usagePercent‚70.51396025655976 disk.drbd1817.out‚0 disk.sda.out‚ 785.06640625 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 785.06640625 disk.rbd1.out‚93.03125 disk.nvme0n1.in‚ 103.59375 disk.drbd1226.in‚0* proc.memory.percent‚11.843840281168621% disk./.usageGB‚1260.8612251281738 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1946.0078125 network.recv‚ 1.2985e+06 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0# memory_percent‚92.0816739400228 disk.dm-0.in‚ 103.59375# disk.dm-0.out‚47119.09619140625& disk.nvme0n1.out‚47119.09619140625 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0" proc.memory.rssMB‚ 2910.7421875 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0vÉ»c”í*‚(:‰ 憋ÓÄز³ gpu.0.gpu‚51.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚316.413' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚105.471 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿’5E±î*‚(:¹  憋ÓÌÂÖ³ disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.sda.out‚ 785.39453125 disk.sda2.in‚ 0.03515625" proc.memory.rssMB‚ 2910.7421875* proc.memory.percent‚11.843840281168621 disk.sda.in‚ 0.03515625 disk.drbd2078.out‚0 disk.sda2.out‚ 785.39453125# memory_percent‚92.0816739400228 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd15‡•’”“50.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 network.recv‚ 1.300534e+06( disk./.usagePercent‚70.5139579098435 disk.drbd1356.out‚0% disk.nvme0n1.out‚47119.5146484375 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1285.in‚0 network.sent‚ 1.755419e+07$ disk./.usageGB‚1260.861183166504 disk.nvme0n1.in‚ 103.59375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0( proc.memory.availableMB‚ 1946.0078125 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚93.03125 disk.dm-0.in‚ 103.59375" disk.dm-0.out‚47119.5146484375 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0 cpu‚25.335575855567345ÙGÖ<žï*‚(:“ õ†‹Óì·Ð³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚303.56' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.18666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$§§ˆJð*‚(:¿  õ†‹Ó¾Ñë³ disk.rbd1.out‚93.09375* proc.memory.percent‚11.843840281168621 proc.cpu.threads‚17) disk./.usagePercent‚70.51383929400399 disk.drbd1673.in‚0 disk.dm-0.in‚ 103.77734375 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 103.77734375 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 cpu‚25.335505076206843 network.sent‚ 1.755516e+07 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0) proc.memory.availableMB‚ 1945.74609375 disk.sda1.in‚0 disk.sda.out‚ 785.796875 disk.drbd1289.in‚0 disk.drbd2078.out‚0$ memory_percent‚92.08273887634277 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1226.out‚0" proc.memory.rssMB‚ 2910.7421875 disk.drbd1931.out‚0" disk.dm-0.out‚47119.7412109375 disk.rbd1.in‚0 network.recv‚ 1.301577e+06 disk.drbd2111.out‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚47119.7412109375 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 785.796875 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0% disk./.usageGB‚1260.8590621948242 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0F%Ìžñ*‚(:“ „‡‹Ó€ÛÀ³ gpu.0.gpu‚96.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚244.286' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.42866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TýÖ^Nò*‚(:à  „‡‹Óß‘ß³ disk.drbd1584.in‚0 disk.drbd1550.out‚0 network.sent‚ 1.7557612e+07 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚47121.1259765625 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.08273887634277) disk./.usagePercent‚70.51383972067967 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 cpu‚25.33546255207076 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1931.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 786.36328125 disk.drbd1938.out‚0 disk.drbd1285.in‚0" proc.memory.rssMB‚ 2910.7421875 network.recv‚ 1.302593e+06 disk.sda2.out‚ 786.36328125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0" disk.dm-0.out‚47121.1259765625 disk.drbd2118.in‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 103.77734375 disk.nvme0n1.in‚ 103.77734375% disk./.usageGB‚1260.8590698242188 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1945.74609375 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚93.09375* proc.memory.percent‚11.843840281168621?cÒÝó*‚(:’ “‡‹Ó°Ý³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.425' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.141666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ià;@Pô*‚(:Å  “‡‹ÓèÅô³ disk.drbd1584.in‚0 disk.sda.out‚ 786.828125 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.rbd1.out‚ 93.2197265625 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 786.828125 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51383972067967 disk.nvme0n1.in‚ 103.77734375 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1945.74609375 disk.dm-0.in‚ 103.77734375 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1673.in‚0 network.recv‚ 1.303563e+06" proc.memory.rssMB‚ 2910.7421875% disk./.usageGB‚1260.8590698242188 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.sda2.in‚ 0.03515625$ memory_percent‚92.08273887634277 cpu‚25.335405625686384 disk.drbd1552.in‚0 network.sent‚ 1.7560649e+07 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1289.out‚0" disk.dm-0.out‚47121.2939453125 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.843840281168621 disk.drbd1356.in‚0% disk.nvme0n1.out‚47121.2939453125 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚041 .Sõ*‚(:È  ¢‡‹Óôõʳ disk.drbd1550.in‚0 disk.drbd1584.in‚0" proc.memory.rssMB‚ 2910.7421875 cpu‚25.335336575055354 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.843840281168621 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚47122.69873046875# disk.dm-0.out‚47122.69873046875( proc.memory.availableMB‚ 1945.7421875 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 103.77734375( disk./.usagePercent‚70.5138410007067 disk.rbd1.in‚0 disk.drbd1226.out‚0 network.sent‚ 1.7563754e+07 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1965.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 network.recv‚ 1.304658e+06 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 103.77734375 disk.sda.in‚ 0.03515625 disk.sda.out‚ 787.28515625 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 93.3994140625# memory_percent‚92.0827547709147% disk./.usageGB‚1260.8590927124023 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 787.28515625½’üšžö*‚(:“ ¢‡‹Óꥮ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.557' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.18566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ªÓú4÷*/ _wandbruntime‚18270  _runtime‚18270[RkÈø* & lr-AdamW/pg2‚2.237606931499638e-05 trainer/global_step‚10049 $ _timestamp‚1.784857506893398e+09 _runtime‚18270.250628826 _step‚835 & lr-AdamW/pg1‚2.237606931499638e-05[R Íù*Ç & lr-AdamW/pg1‚2.237606931499638e-05 & lr-AdamW/pg2‚2.237606931499638e-05 trainer/global_step‚10049 $ _timestamp‚1.784857506893398e+09 _runtime‚18270.250628826 _step‚835˶I4ú*/ _wandbruntime‚18270  _runtime‚18270ò÷Ú(û*‹P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2903236674316288 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2903236674316288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10485189891584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2902077267830203 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10468977979713 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚383 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93298850 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2892750138988544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚804523753656832 _step‚836 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180521728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2892750112815104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2903235302706688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7848575068943436e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2892750138988544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12824178396160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87222878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2891609634461776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180531798 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93308914 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791699523993600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12824229663232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30613420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30613037 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26777923 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10485189891584 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2903235302706688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10486535327744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93308914 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚791699155902464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87222884 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚383 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93298850 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2892750138988544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚804523753656832 _step‚836 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180521728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2892750112815104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2903235302706688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7848575068943436e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2892750138988544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12824178396160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87222878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2891609634461776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180531798 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93308914 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791699523993600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12824229663232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30613420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30613037 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26777923 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10485189891584 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2903235302706688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10486535327744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93308914 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚791699155902464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87222884 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2893029022105600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2893029022105600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10486212399616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10487557842944 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26780568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚804603585800704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚10049 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87231245 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10469998781465 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.pea®X/k‚ 1728053248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57396584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚804603160158208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180549261 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚385 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚380 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2902357090405619 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93318010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87231251 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10468654170067 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87231251 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180539177 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12825417889792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10486212399616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2903516606121984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2903516606121984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93307932 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚791777742268416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93307932 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2891888461663520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2893029048279040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93318010 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚791778110359552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848575083717468e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180549261 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2902358460444985 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2893029048279040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87231245 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12825475441152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2903515234505216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30616021 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57396584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚804603160158208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180549261 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚385 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚380 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2902357090405619 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93318010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87231251 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10468654170067 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87231251 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180539177 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12825417889792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10486212399616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2903516606121984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2903516606121984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93307932 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚791777742268416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93307932 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2891888461663520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2893029048279040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93318010 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚791778110359552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848575083717468e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180549261 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2902358460444985 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2893029048279040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87231245 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12825475441152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2903515234505216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30616021 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2893029022105600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2893029022105600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10486212399616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10487557842944 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26780568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚804603585800704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚10049Å)ÕØä4€+/ _wandbruntime‚18271  _runtime‚18271ôÁ9  +† /loss/FBt2/detection_loss‚0.4428422955042363 &loss/total_loss‚7.8038864441839575 _step‚838 2loss/SRqqt1/assignment_loss‚0.5310609441929552 3loss/SRqqt2/assignment_loss‚0.47557667275344623 /loss/FRt1/detection_loss‚0.5392582110916526 % _timestamp‚1.7848575084057858e+09 1loss/FBt1/assignment_loss‚0.01256629210450666 /loss/FRt2/detection_loss‚1.4992085113702343 1loss/SRqqt2/detection_loss‚0.5903876103208284 epoch‚35 trainer/global_step‚10049 1loss/FBt2/assignment_loss‚0.01792929620550409 1loss/SRqqt1/detection_loss‚0.8765640959723807 _runtime‚18271.763065019 0loss/FRt1/assignment_loss‚1.5471151689287566 0loss/FRt2/assignment_loss‚0.7590828505158425 /loss/FBt1/detection_loss‚0.5122944952236139„È—Ý‚+‹ 1loss/FBt2/assignment_loss‚0.01792929620550409 1loss/SRqqt1/detection_loss‚0.8765640959723807 _runtime‚18271.763065019 0loss/FRt1/assignment_loss‚1.5471151689287566 0loss/FRt2/assignment_loss‚0.7590828505158425 /loss/FBt1/detection_loss‚0.5122944952236139 /loss/FBt2/detection_loss‚0.4428422955042363 &loss/total_loss‚7.8038864441839575 _step‚838 2loss/SRqqt1/assignment_loss‚0.5310609441929552 3loss/SRqqt2/assignment_loss‚0.47557667275344623 /loss/FRt1/detection_loss‚0.5392582110916526 % _timestamp‚1.7848575084057858e+09 1loss/FBt1/assignment_loss‚0.01256629210450666 /loss/FRt2/detection_loss‚1.4992085113702343 1loss/SRqqt2/detection_loss‚0.5903876103208284 epoch‚35 trainer/global_step‚10049ÆaŒÆRƒ+‚(:Ç  ±‡‹Ó±¾…´ disk.drbd1510.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.sda1.out‚0$ memory_percent‚92.08013216654459& disk.nvme0n1.out‚47122.90185546875 disk.drbd1226.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1946.38671875 network.sent‚ 1.7566796e+07 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 93.4619140625 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0" proc.memory.rssMB‚ 2910.7421875 cpu‚25.33527902610334% disk./.usageGB‚1260.8590927124023 disk.nvme0n1.in‚ 105.9609375 disk.sda1.in‚0 disk.sda.out‚ 787.80078125 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 network.recv‚ 1.305693e+06 disk.drbd1965.in‚0 disk.drbd1584.out‚0( disk./.usagePercent‚70.5138410007067 disk.dm-0.in‚ 105.9609375# disk.dm-0.out‚47122.90185546875 disk.drbd1550.out‚0 disk.sda2.out‚ 787.80078125 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.843840281168621¢~.mž„+‚(:“ ±‡‹Ó‘“æ³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚254.631' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.87700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XzT1S…+‚(:È  À‡‹ÓïõÙ³ disk.drbd1285.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.08013216654459 disk.drbd1965.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1946.38671875 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17" proc.memory.rssMB‚ 2910.7421875 disk.rbd1.in‚0 disk.rbd1.out‚ 93.5478515625 disk.drbd1226.in‚0 disk.drbd1510.in‚0 network.sent‚ 1.762678e+07% disk./.usageGB‚1260.8591117858887 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 788.18359375 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.843840281168621) disk./.usagePercent‚70.51384206739591 disk.sda1.in‚0 disk.sda2.out‚ 788.18359375# disk.dm-0.out‚47123.36083984375 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 network.recv‚ 1.307661e+06 disk.sda1.out‚0 disk.drbd1938.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 105.9609375& disk.nvme0n1.out‚47123.36083984375 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 cpu‚25.335223869078288 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 105.9609375«_Qe†+‚(:’ À‡‹ÓàÄ·³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚302.572' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.85733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î7Ìhž‡+‚(:“ χ‹ÓÕ»µ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.709' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.90299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04%‘×Oˆ+‚(:Ä  χ‹ÓûÜÓ³ disk.drbd1673.in‚0 disk.drbd1356.out‚0$ memory_percent‚92.07913080851236 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 105.9609375 disk.drbd1965.out‚0* proc.memory.percent‚11.843840281168621 disk.sda1.out‚0# disk.dm-0.out‚47123.55224609375 cpu‚25.335167965896233 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚47123.55224609375 disk.drbd1510.out‚0" proc.memory.rssMB‚ 2910.7421875 network.sent‚ 1.762983e+07 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 788.5078125 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0% disk./.usageGB‚1260.8590927124023 disk.nvme0n1.in‚ 105.9609375 disk.drbd1226.out‚0 network.recv‚ 1.308739e+06 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0( proc.memory.availableMB‚ 1946.6328125 disk.drbd1931.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 788.5078125 disk.drbd1828.in‚0( disk./.usagePercent‚70.5138410007067 disk.drbd2118.in‚0 disk.rbd1.out‚ 93.7041015625 proc.cpu.threads‚17f ËâU‰+‚(:Ê  Þ‡‹ÓÇÔ³ disk.rbd1.out‚ 93.7041015625 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 1940.88671875* proc.memory.percent‚11.865266164143879 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 network.recv‚ 1.309675e+06( disk./.usagePercent‚70.5138410007067 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.10251172383627 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 106.52734375 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1510.out‚0 network.sent‚ 1.7633379e+07 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0" proc.memory.rssMB‚ 2916.0078125 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.sda.out‚ 788.87109375 cpu‚25.335874379262425 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0% disk./.usageGB‚1260.8590927124023 disk.drbd2078.out‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚47123.88720703125 disk.dm-0.in‚ 106.52734375 disk.drbd2041.in‚0 disk.sda2.out‚ 788.87109375# disk.dm-0.out‚47123.88720703125ÆåaŸŸŠ+‚(:” Þ‡‹Ó‰Ó¶³ gpu.0.gpu‚74.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.467' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.48899999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Z†4‹+/ _wandbruntime‚18343  _runtime‚18343ìTwbŒ+ôu 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3848581612110138 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8404452204704285 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6808657050132751 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3659570813179016 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36504390835762024 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8093236088752747 +jet/accuracy_3_of_3‚0.07591768354177475 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3580916225910187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04049088433384895 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7952157258987427 *particle/sensitivity‚0.641785740852356 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.658241331577301 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2963905930519104 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1758354753255844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6271960139274597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 )jet/accuracy_2_of_4‚0.847897469997406 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3454962372779846 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025380264967679977 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7826274633407593 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46987298130989075 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7190603017807007 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3809843361377716 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6367890238761902 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7556814551353455 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9407419562339783 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 _runtime‚18343.167070448 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8042339086532593 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03106306679546833 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2934381365776062 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5271416902542114 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6230769753456116 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6577986478805542 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17015239596366882 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.879010796546936 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7241509556770325 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7766826748847961 )jet/accuracy_1_of_2‚0.785550594329834 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚10079 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.831516683101654 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36777493357658386 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7982374429702759 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6320415735244751 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8701725006103516 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6023291349411011 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014799253083765507 particle/accuracy_1_of_2‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7068577408790588 7 Purity/*FRt0SR/ý§šø%qqt1FBt/FRt_purity‚0.7597553133964539 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9993678331375122 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26768025755882263 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_1_of_1‚0.36777034401893616 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6842250227928162 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28965234756469727 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27636459469795227 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4655487537384033 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4036106467247009 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024798991158604622 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8804399371147156 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7148045897483826 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 (particle/accuracy‚0.8532539010047913 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6347827911376953 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7156485319137573 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18774643540382385 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.30774015188217163 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.049895044416189194 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03188321739435196 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6500505208969116 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7398441433906555 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35402292013168335 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.879010796546936 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.789809065211563 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09054268151521683 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 *jet/accuracy_1_of_3‚0.9356136322021484 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8392395377159119 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4613973796367645 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7777159810066223 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.6133125424385071 7validation_average_jet_accuracy‚0.40260839462280273 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6337406039237976 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7238406538963318 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9110174775123596 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3870656490325928 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9220250248908997 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6207599639892578 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27743881940841675 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5496402978897095 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚839 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.537564218044281 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22931955754756927 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6367890238761902 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 epoch‚35 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7149016261100769 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014799253083765507 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35402292013168335 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7725054025650024 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8308854103088379 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1807795614004135 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 % _timestamp‚1.7848575798089666e+09 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.630455493927002 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3773954510688782 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2694289982318878 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2851737439632416 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6998456120491028 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2329089641571045 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6531170010566711 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7959776520729065 /particle/accuracy_4_of_4‚0.9716639518737793 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.687049925327301 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6736952662467957 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1484900861978531 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6612197756767273 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15797634422779083 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07356662303209305 *jet/accuracy_1_of_4‚0.9709590673446655 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3057221472263336 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6855944991111755 +jet/accuracy_2_of_2‚0.17321494221687317 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35186484456062317 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6328807473182678 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 'particle/f_score‚0.6523555517196655 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35443827509880066 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864170432090759 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 +jet/accuracy_4_of_4‚0.02295798622071743 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062‹Iÿ:+ùu =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09054268151521683 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 *jet/accuracy_1_of_3‚0.9356136322021484 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8392395377159119 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4613973796367645 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7777159810066223 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.6133125424385071 7validation_average_jet_accuracy‚0.40260839462280273 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6337406039237976 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7238406538963318 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9110174775123596 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3870656490325928 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9220250248908997 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6207599639892578 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27743881940841675 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5496402978897095 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚839 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.537564218044281 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22931955754756927 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6367890238761902 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 epoch‚35 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7149016261100769 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014799253083765507 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35402292013168335 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7725054025650024 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8308854103088379 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1807795614004135 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 % _timestamp‚1.7848575798089666e+09 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.630455493927002 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3773954510688782 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2694289982318878 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2851737439632416 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6998456120491028 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2329089641571045 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6531170010566711 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7959776520729065 /particle/accuracy_4_of_4‚0.9716639518737793 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.687049925327301 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6736952662467957 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1484900861978531 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6612197756767273 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15797634422779083 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07356662303209305 *jet/accuracy_1_of_4‚0.9709590673446655 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3057221472263336 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6855944991111755 +jet/accuracy_2_of_2‚0.17321494221687317 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35186484456062317 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6328807473182678 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 'particle/f_score‚0.6523555517196655 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35443827509880066 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864170432090759 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 +jet/accuracy_4_of_4‚0.02295798622071743 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3848581612110138 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8404452204704285 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6808657050132751 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3659570813179016 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36504390835762024 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8093236088752747 +jet/accuracy_3_of_3‚0.07591768354177475 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3580916225910187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04049088433384895 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7952157258987427 *particle/sensitivity‚0.641785740852356 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.658241331577301 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2963905930519104 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1758354753255844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6271960139274597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 )jet/accuracy_2_of_4‚0.847897469997406 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3454962372779846 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025380264967679977 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7826274633407593 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46987298130989075 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7190603017807007 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3809843361377716 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6367890238761902 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7556814551353455 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9407419562339783 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 _runtime‚18343.167070448 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8042339086532593 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03106306679546833 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2934381365776062 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5271416902542114 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6230769753456116 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6577986478805542 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17015239596366882 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.879010796546936 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7241509556770325 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7766826748847961 )jet/accuracy_1_of_2‚0.785550594329834 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚10079 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.831516683101654 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36777493357658386 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7982374429702759 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6320415735244751 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8701725006103516 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6023291349411011 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014799253083765507 particle/accuracy_1_of_2‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7068577408790588 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7597553133964539 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9993678331375122 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26768025755882263 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_1_of_1‚0.36777034401893616 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6842250227928162 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28965234756469727 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27636459469795227 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4655487537384033 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4036106467247009 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024798991158604622 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8804399371147156 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7148045897483826 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 (particle/accuracy‚0.8532539010047913 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6347827911376953 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7156485319137573 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18774643540382385 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.30774015188217163 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.049895044416189194 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03188321739435196 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6500505208969116 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7398441433906555 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35402292013168335 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.879010796546936 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.789809065211563 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1ÇÅŒp TŽ+‚(:É  퇋ÓÎéè³ network.sent‚ 1.7635842e+07% disk./.usageGB‚1260.8590927124023 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚47124.06689453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 789.2265625 disk.rbd1.in‚0 disk.drbd1931.out‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2921.27734375 disk.nvme0n1.in‚ 106.52734375 disk.sda2.out‚ 789.2265625* proc.memory.percent‚11.886707941691082 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0$ memory_percent‚92.12295214335123( disk./.usagePercent‚70.5138410007067 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 106.52734375) proc.memory.availableMB‚ 1935.86328125 disk.drbd1828.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0# disk.dm-0.out‚47124.06689453125 disk.drbd1965.out‚0 network.recv‚ 1.310635e+06 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 93.7666015625 disk.drbd1550.out‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 cpu‚25.337681008651938 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0Î_+öŸ+‚(:” 퇋Óö¹Ì³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚304.361' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.45366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éA “+‚(:ˆ ü‡‹Ó•´Æ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚285.837' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.279 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ütÝT‘+‚(:É  ü‡‹Óý‰é³ disk.drbd1673.in‚0# proc.memory.rssMB‚ 2921.27734375 network.sent‚ 1.7689703e+07% disk./.usageGB‚1260.8590927124023 disk.drbd2041.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1285.out‚0( disk./.usagePercent‚70.5138410007067 disk.rbd1.out‚ 93.7666015625 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 789.61328125$ memory_percent‚92.12385813395181 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 106.52734375' proc.memory.availableMB‚ 1935.640625 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0* proc.memory.percent‚11.886707941691082 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 cpu‚25.337650271006787# disk.dm-0.out‚47124.40673828125 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda.out‚ 789.61328125 disk.drbd1931.out‚0 disk.dm-0.in‚ 106.52734375 disk.drbd1550.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚47124.40673828125 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 network.recv‚ 1.312393e+06 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd1584.in‚09LPÑž’+‚(:“ ‹ˆ‹ÓÀ¼³ gpu.0.gpu‚83.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.803' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.26766666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽK³S“+‚(:È  ‹ˆ‹Ó‡¾â³ disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 789.98828125 disk.drbd1510.in‚0& disk.nvme0n1.out‚47125.15283203125 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8650283813477 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 cpu‚25.33764690439324 disk.drbd1552.in‚0 network.recv‚ 1.313437e+06 disk.rbd1.out‚ 93.892578125) disk./.usagePercent‚70.51417295438708 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.886707941691082 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 106.52734375 disk.drbd1965.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.12449391682944 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2921.27734375 disk.drbd1584.out‚0 disk.drbd1226.in‚0' proc.memory.availableMB‚ 1935.484375 disk.sda2.out‚ 789.98828125 disk.drbd2118.out‚0 disk.drbd1931.out‚0# disk.dm-0.out‚47125.15283203125 network.sent‚ 1.7692713e+07 disk.drbd1817.in‚0…4”+/ _wandbruntime‚18376  _runtime‚18376D™†Ë•+Å ' lr-AdamW/pg2‚2.1851278495997383e-05 trainer/global_step‚10099 % _timestamp‚1.7848576129542773e+09 _runtime‚18376.311546076 _step‚840 ' lr-AdamW/pg1‚2.1851278495997383e-05W¥Ð–+Ê ' lr-AdamW/pg1‚2.1851278495997383e-05 ' lr-AdamW/pg2‚2.1851278495997383e-05 trainer/global_step‚10099 % _timestamp‚1.7848576129542773e+09 _runtime‚18376.311546076 _step‚840Èð—5‡4—+/ _wandbruntime‚18376  _runtime‚18376Š ªà˜+ŠP WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57705221 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26916820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87730660 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10541997884416 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCD%UÃ60#eviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87730660 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2918634440431542 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2918633070392288 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2919799623558656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚40781312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚808571276254720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87730666 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181542839 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30788401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 408872448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93802109 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2909256280238080 trainer/global_step‚10099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26916815 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2909256280238080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93802109 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _runtime‚18376.315934374 _step‚841 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚808571685127168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181532769 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57704910 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10543343320576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12893802541568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87730666 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2909256254064640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚306 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784857612955386e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚311 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2919799623558656 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2919798251949056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10543343320576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12893843322880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10524346090544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10541997884416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93812173 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2909256280238080 trainer/global_step‚10099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26916815 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2909256280238080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93802109 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 _runtime‚18376.315934374 _step‚841 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚808571685127168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181532769 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57704910 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10543343320576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12893802541568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87730666 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2909256254064640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚306 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784857612955386e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚311 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2919799623558656 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2919798251949056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10543343320576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12893843322880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10524346090544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10541997884416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93812173 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚808651125928448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12895108720640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93821269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚795756060079104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42871296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10544365835776 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚ 18377.7900765 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚314 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181560302 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚808651536890880 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12895065849344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93811191 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2909535189528576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2909535163355136 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10525366892184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93811191 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2918912892967704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10544365835776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87739027 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2909535189528576 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2908387526075520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10526711503582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93821269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181550218 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26919455 trainer/global_step‚10099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚795756428170240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30791373 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57710519 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57710833 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚309 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87739033 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10543020392448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2920078183747584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93821269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181550218 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26919455 trainer/global_step‚10099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚795756428170240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30791373 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57710519 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57710833 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚309 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87739033 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10543020392448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2920078183747584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚808651125928448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12895108720640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93821269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚795756060079104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42871296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10544365835776 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚ 18377.7900765 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚314 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181560302 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚808651536890880 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12895065849344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93811191 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2909535189528576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2909535163355136 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10525366892184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112ÊBF¦Ç4+/ _wandbruntime‚18377  _runtime‚18377«·vž+‰ 2loss/FBt2/assignment_loss‚0.022385123701525202 /loss/FRt1/detection_loss‚0.5310050610956899 % _timestamp‚1.7848576144684396e+09 _runtime‚18377.825753208 _step‚843 3loss/FBt1/assignment_loss‚4.197641487481617e-06 /loss/FBt2/detection_loss‚0.3726235302804298 epoch‚36 0loss/FRt2/assignment_loss‚0.6351711915898647 3loss/SRqqt1/assignment_loss‚0.45133578050374934 3loss/SRqqt2/assignment_loss‚0.44893401202221295 1loss/SRqqt2/detection_loss‚0.5922197081864915 trainer/global_step‚10099 .loss/FRt2/detection_loss‚1.364799571029707 1loss/SRqqt1/detection_loss‚0.9104875370263471 /loss/FBt1/detection_loss‚0.8224637941789946 &loss/total_loss‚7.7496185062969625 0loss/FRt1/assignment_loss‚1.5981889990404634žXŸ+Ž 1loss/SRqqt2/detection_loss‚0.5922197081864915 trainer/global_step‚10099 .loss/FRt2/detection_loss‚1.364799571029707 1loss/SRqqt1/detection_loss‚0.9104875370263471 /loss/FBt1/detection_loss‚0.8224637941789946 &loss/total_loss‚7.7496185062969625 0loss/FRt1/assignment_loss‚1.5981889990404634 2loss/FBt2/assignment_loss‚0.022385123701525202 /loss/FRt1/detection_loss‚0.5310050610956899 % _timestamp‚1.7848576144684396e+09 _runtime‚18377.825753208 _step‚843 3loss/FBt1/assignment_loss‚4.197641487481617e-06 /loss/FBt2/detection_loss‚0.3726235302804298 epoch‚36 0loss/FRt2/assignment_loss‚0.6351711915898647 3loss/SRqqt1/assignment_loss‚0.45133578050374934 3loss/SRqqt2/assignment_loss‚0.44893401202221295Ëب/ÛS +‚(:È  šˆ‹Ó•³‚´ disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 790.29296875 disk.drbd1938.out‚0% disk./.usageGB‚1260.8659744262695 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 106.52734375 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1931.out‚0# disk.dm-0.out‚47125.53466796875 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2921.27734375 disk.drbd1817.out‚0 disk.sda.out‚ 790.29296875 cpu‚25.337574831400904 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚47125.53466796875 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 network.sent‚ 1.7695708e+07 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.886707941691082 network.recv‚ 1.31453e+06( disk./.usagePercent‚70.5142258621716 disk.drbd1938.in‚0 disk.dm-0.in‚ 106.52734375 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.12681452433267 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.rbd1.out‚ 93.892578125( proc.memory.availableMB‚ 1934.9140625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0×®T.“¡+‚(:ˆ šˆ‹ÓÙ›ä³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚277.371' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.457 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽñÖ½T¢+‚(:É  ©ˆ‹ÓÈóö³ network.recv‚ 1.316328e+06 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.886707941691082) disk./.usagePercent‚70.51422031538775 disk.drbd1510.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.12689399719238) proc.memory.availableMB‚ 1934.89453125 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 790.5703125& disk.nvme0n1.out‚47126.25732421875 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 93.955078125 disk.drbd1931.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 106.52734375 disk.sda1.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd2118.out‚0 network.sent‚ 1.7755785e+07# proc.memory.rssMB‚ 2921.27734375 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.sda1.out‚0% disk./.usageGB‚1260.8658752441406 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.sda2.out‚ 790.5703125 cpu‚25.337544512469094 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1285.in‚0# disk.dm-0.out‚47126.25732421875 disk.drbd2078.out‚0 disk.drbd1356.in‚0K{ùµŸ£+‚(:” ©ˆ‹Óä›Ù³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚310.493' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.49766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÜ‘ûž¤+‚(:“ ¸ˆ‹Ó™êÞ³ gpu.0.gpu‚68.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.592' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.19733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ƳEO¥+‚(:Ä  ¸ˆ‹Ó¯…ÿ³ disk.drbd1584.out‚0 disk.nvme0n1.in‚ 106.52734375# memory_percent‚92.1269416809082 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.sda2.in‚ 0.03515625( proc.memory.availableMB‚ 1934.8828125 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 106.52734375 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.sda.in‚ 0.03515625 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 network.recv‚ 1.317292e+06 cpu‚25.33748672264477 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51418660800888 disk.drbd1817.in‚0" disk.dm-0.out‚47127.3486328125 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.886707941691082 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 790.9140625% disk./.usageGB‚1260.8652725219727 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2078.out‚0% disk.nvme0n1.out‚47127.3486328125 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2921.27734375 disk.sda.out‚ 790.9140625 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 93.955078125 network.sent‚ 1.7758783e+07 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0;bÌŸ¦+‚(:” Lj‹ÓØ帳 gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚312.925' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.30833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë9”›T§+‚(:É  Lj‹Óø‚Ú³ disk.rbd1.out‚ 94.0810546875 disk.drbd1584.in‚0 disk.drbd2078.in‚0% disk.nvme0n1.out‚47127.8212890625 disk.drbd1226.out‚0 disk.sda2.in‚ 0.03515625( proc.memory.availableMB‚ 1934.7421875) disk./.usagePercent‚70.51418660800888 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 106.52734375 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 791.20703125 disk.drbd1938.out‚0 cpu‚25.337416490239907 disk.drbd1510.in‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1289.out‚0 network.sent‚ 1.7761895e+07 disk.drbd1285.out‚0 disk.drbd1938.in‚0$ memory_percent‚92.12751388549805 disk.drbd1828.out‚0 disk.drbd1817.in‚0# proc.memory.rssMB‚ 2921.27734375 network.recv‚ 1.318413e+06 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0% disk./.usageGB‚1260.8652725219727" disk.dm-0.out‚47127.8212890625 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.sda2.out‚ 791.20703125 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd2111.out‚0 disk.drbd1965.out‚0 disk.sda1.in‚0* proc.memory.percent‚11.886707941691082 disk.drbd2078.out‚0 disk.drbd1828.in‚0ÁËÚ7ž¨+‚(:“ Öˆ‹Ó‰™¾³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.613' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.53766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤°N£U©+‚(:Ê  Öˆ‹ÓøÕà³ disk.rbd1.in‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd1550.in‚0% disk./.usageGB‚1260.8652725219727 disk.drbd1584.out‚0 cpu‚25.337372457087017 disk.sda2.out‚ 791.6015625 disk.dm-0.in‚ 106.52734375 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0$ memory_percent‚92.12752978006998 disk.drbd1285.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17 network.recv‚ 1.319535e+06 disk.sda.out‚ 791.6015625 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2921.27734375 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1934.73828125& disk.nvme0n1.out‚47128.72021484375 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625# disk.dm-0.out‚47128.72021484375 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 network.sent‚ 1.7765484e+07 disk.drbd1965.out‚0 disk.rbd1.out‚ 94.0810546875 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.886707941691082 disk.drbd2111.out‚0) disk./.usagePercent‚70.51418660800888 disk.drbd2078.in‚0 disk.drbd2078.out‚0}ÀŠ4ª+/ _wandbruntime‚18463  _runtime‚18463å N<Ç«+Á % lr-AdamW/pg1‚2.13312999407059e-05 % lr-AdamW/pg2‚2.13312999407059e-05 trainer/global_step‚10149 % _timestamp‚1.7848576999693456e+09 _runtime‚18463.326647781 _step‚844&Î7â̬+Æ trainer/global_step‚10149 % _timestamp‚1.7848576999693456e+09 _runtime‚18463.326647781 _step‚844 % lr-AdamW/pg1‚2.13312999407059e-05 % lr-AdamW/pg2‚2.13312999407059e-05ÌG_ë‘4­+/ _wandbruntime‚18463  _runtime‚18463×F(®+‹P O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27048820 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182407071 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88149010 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182417141 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94268125 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚799607160101888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2933796277639680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2923201718589440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _runtime‚18463.328073768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2933796277639680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2922048838418512 trainer/global_step‚10149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88149016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10594532876800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94258061 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚845 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94268125 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2932624262963896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94258061 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2933794906030080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88149016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30935764 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10576794584638 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30936103 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544‹¾ÿ;’ ¯+P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12956560278528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10575449973352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2923201718589440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2933794906030080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2932625633003150 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2922048838418512 trainer/global_step‚10149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88149016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10594532876800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94258061 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/in*ØŠ„active_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚845 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94268125 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2932624262963896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94258061 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2933794906030080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88149016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30935764 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10576794584638 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30936103 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27048820 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚655 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182407071 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88149010 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182417141 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94268125 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚799607160101888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2933796277639680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2923201718589440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _runtime‚18463.328073768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2933796277639680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88157383 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚338 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _step‚846 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10595555392000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2934074837828608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2923480627879936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57990155 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚799685746467840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10594209948672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88157377 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10594209948672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88157383 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94277221 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2922327640192288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10577815386390 trainer/global_step‚10149 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2934076209445376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2923480654053376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88157377 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2932904085539312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚812643547447296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2932905455578678 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2922327614764320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182424520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182434604 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10595555392000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _runtime‚18464.803575233 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88157383 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚338 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _step‚846 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10595555392000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2934074837828608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2923480627879936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57990155 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚799685746467840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10594209948672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88157377 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10594209948672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88157383 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94277221 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2922327640192288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10577815386390 trainer/global_step‚10149 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2934076209445376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2923480654053376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88157377 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2932904085539312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚812643547447296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2932905455578678 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2922327614764320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182424520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182434604 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10595555392000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _runtime‚18464.803575233 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88567366 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94724077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58263038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10626553856160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31082218 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2937147209287680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚816555256906240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2946615455535504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183281373 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10645722433024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚816554827062272 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2935988901679344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10627898467446 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚803536110308352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2937147183114240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10644376996864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88567366 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2947792931720704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10645722433024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚18548.859993207 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2947791560111104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2946616825574758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27180815 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183291443 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2947791560111104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2937147209287680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚803536478399488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94714013 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88567360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 _step‚849UR(Ç+P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2946615455535504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183281373 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10645722433024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚816554827062272 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2935988901679344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10627898467446 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚803536110308352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2937147183114240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10644376996864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88567366 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2947792931720704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10645722433024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 _runtime‚18548.859993207 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2947791560111104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2946616825574758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27180815 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183291443 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2947791560111104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2937147209287680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚803536478399488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94714013 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88567360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚623 _step‚849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88567360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94714013 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88567366 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94724077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58263038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10626553856160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31082218 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2937147209287680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚816555256906240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160Ñ9ÇÓ4È+/ _wandbruntime‚18550  _runtime‚18550ªG°È2É+•N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88575727 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚18550.331470188 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94723095 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27183455 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2936267703453120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10628919269198 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88575727 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚850 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚803615064765440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58268219 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183298822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31084764 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88575733 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183308906 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94733173 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚803614696674304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10645399504896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2948071491909632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94723095 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2948072863526400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚816634653702144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848577869739165e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10646744948224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2937426092404736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10645399504896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183308906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94733173 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29362677288810886U^T 'Ê+šN TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94723095 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27183455 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2936267703453120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10628919269198 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88575727 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 _step‚850 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚803615064765440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58268219 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183298822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31084764 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88575733 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183308906 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94733173 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚803614696674304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10645399504896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2948071491909632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94723095 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2948072863526400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚816634653702144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848577869739165e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10646744948224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2937426092404736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10645399504896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183308906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94733173 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2936267728881088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88575727 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚18550.331470188 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112ÒÍÕ5V4Ë+/ _wandbruntime‚18550  _runtime‚18550º7en Ì+† /loss/FRt2/detection_loss‚1.4009524890967946 trainer/global_step‚10199 2loss/FBt2/assignment_loss‚0.016821469731366598 /loss/FRt1/detection_loss‚0.5225890641457407 1loss/SRqqt1/detection_loss‚0.8343705004680965 1loss/SRqqt2/detection_loss‚0.5802357324141733 _step‚851 0loss/FRt1/assignment_loss‚1.6631541266889855 2loss/SRqqt1/assignment_loss‚0.5797673651347997 %loss/total_loss‚7.617293553236894 _runtime‚18550.371470021 0loss/FRt2/assignment_loss‚0.6350148618879171 2loss/SRqqt2/assignment_loss‚0.4689993885278785 2loss/FBt1/assignment_loss‚0.006990626517133245 /loss/FBt1/detection_loss‚0.5030442480570995 /loss/FBt2/detection_loss‚0.4053536805669082 epoch‚36 % _timestamp‚1.7848577870142684e+09yÑ2RÍ+‹ %loss/total_loss‚7.617293553236894 _runtime‚18550.371470021 0loss/FRt2/assignment_loss‚0.6350148618879171 2loss/SRqqt2/assignment_loss‚0.4689993885278785 2loss/FBt1/assignment_loss‚0.006990626517133245 /loss/FBt1/detection_loss‚0.5030442480570995 /loss/FBt2/detection_loss‚0.4053536805669082 epoch‚36 % _timestamp‚1.7848577870142684e+09 /loss/FRt2/detection_loss‚1.4009524890967946 trainer/global_step‚10199 2loss/FBt2/assignment_loss‚0.016821469731366598 /loss/FRt1/detection_loss‚0.5225890641457407 1loss/SRqqt1/detection_loss‚0.8343705004680965 1loss/SRqqt2/detection_loss‚0.5802357324141733 _step‚851 0loss/FRt1/assignment_loss‚1.6631541266889855 2loss/SRqqt1/assignment_loss‚0.5797673651347997Ó‰!MÎ+‚(:  ¿‰‹Ó¸úõ³ disk.sda2.in‚ 0.03515625 disk.drbd1552.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0* proc.memory.percent‚11.886707941691082 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2921.27734375 network.recv‚ 1.327446e+06% disk./.usageGB‚1260.8626823425293 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 106.52734375 disk.rbd1.out‚ 94.66796875 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 794.16796875 disk.drbd1550.out‚0) disk./.usagePercent‚70.51404175161495 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2118.in‚0! disk.dm-0.out‚47134.759765625$ memory_percent‚92.12671915690103% proc.memory.availableMB‚ 1934.9375 network.sent‚ 1.7843246e+07 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.sda.out‚ 794.16796875 disk.drbd1965.in‚0 cpu‚25.336959140650475 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚47134.759765625 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0.x&žÏ+‚(:“ ¿‰‹Ó—åÚ³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚272.233' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.74433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yœ“Ð+‚(:ˆ Ή‹Ó”ÃÙ³ gpu.0.gpu‚64.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚104.787' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚34.929 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åî)RÑ+‚(:Ç  Ή‹Óúö³ disk.dm-0.in‚ 106.52734375 disk.sda1.in‚0 disk.drbd1226.in‚0) proc.memory.availableMB‚ 1934.69140625 disk.drbd1226.out‚0 disk.drbd1552.out‚0 network.recv‚ 1.329157e+06( disk./.usagePercent‚70.5140404715879 disk.drbd1673.out‚0 disk.sda2.out‚ 794.53515625 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd2111.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 794.53515625 disk.sda2.in‚ 0.03515625 disk.drbd2078.out‚0 disk.drbd2041.in‚0% disk.nvme0n1.out‚47136.8056640625 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2921.27734375 cpu‚25.336915880249688 proc.cpu.threads‚17% disk./.usageGB‚1260.8626594543457 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 network.sent‚ 1.7903203e+07 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.12772051493326 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 94.66796875 disk.nvme0n1.in‚ 106.52734375* proc.memory.percent‚11.886707941691082 disk.drbd1817.in‚0" disk.dm-0.out‚47136.8056640625 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625¥ex[žÒ+‚(:“ ݉‹Ó«œÄ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚291.619' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.20633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ*£LÓ+‚(:Á  ݉‹ÓÛýã³ network.recv‚ 1.330212e+06( disk./.usagePercent‚70.5140394048987 disk.drbd2111.in‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2921.27734375* proc.memory.percent‚11.886707941691082 disk.dm-0.in‚ 106.52734375 disk.drbd1552.out‚0 disk.drbd1817.out‚0$ memory_percent‚92.12772051493326 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 794.84375" disk.dm-0.out‚47137.2744140625 disk.drbd1510.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 94.73046875 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8626403808594 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd2041.in‚0) proc.memory.availableMB‚ 1934.69140625 disk.drbd1828.out‚0 disk.drbd1550.in‚0 cpu‚25.336859708364077 disk.drbd1965.in‚0 disk.drbd2041.out‚0% disk.nvme0n1.out‚47137.2744140625 disk.drbd1584.out‚0 disk.sda.out‚ 794.84375 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 network.sent‚ 1.7906244e+07 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 106.52734375 proc.cpu.threads‚17ÃWXžÔ+‚(:“ 쉋ÓãÕº³ gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.257' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.08566666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó…=ŽTÕ+‚(:É  쉋ÓÿÅÛ³ disk.drbd1673.out‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚47137.83251953125# proc.memory.rssMB‚ 2921.27734375 disk.nvme0n1.in‚ 106.52734375 disk.sda1.in‚0% disk./.usageGB‚1260.8626403808594 disk.dm-0.in‚ 106.52734375 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 94.73046875 disk.drbd1584.out‚0 disk.sda2.out‚ 795.15234375 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0( disk./.usagePercent‚70.5140394048987 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.rbd1.in‚0 cpu‚25.336816805207118 network.recv‚ 1.331129e+06 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0# disk.dm-0.out‚47137.83251953125 disk.drbd2078.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.886707941691082 disk.drbd2118.in‚0 disk.sda.out‚ 795.15234375 disk.drbd1673.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.12772051493326 disk.drbd1552.in‚0) proc.memory.availableMB‚ 1934.69140625 proc.cpu.threads‚17 network.sent‚ 1.7909342e+07 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0»ëÖ+‚(:’ û‰‹Ó™¸á³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.672' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.224000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð_Š±N×+‚(:à  û‰‹ÓÁÐÿ³ disk.sda2.out‚ 795.453125 disk.sda.in‚ 0.03515625 disk.drbd1510.in‚0 cpu‚25.336759574694085 disk.sda2.in‚ 0.03515625# disk.dm-0.out‚47138.32470703125# proc.memory.rssMB‚ 2921.27734375 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd2118.out‚0 disk.sda.out‚ 795.453125 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 94.9423828125 disk.drbd1289.in‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 network.sent‚ 1.7912886e+07 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚47138.32470703125 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0( disk./.usagePercent‚70.5140394048987% disk./.usageGB‚1260.8626403808594 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.sda1.in‚0$ memory_percent‚92.12773640950522 network.recv‚ 1.332137e+06 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 106.52734375 disk.drbd1510.out‚0% proc.memory.availableMB‚ 1934.6875* proc.memory.percent‚11.886707941691082„ °žØ+‚(:“ ŠŠ‹ÓÕ¹ê³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.555' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.51833333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¬ÌRÙ+‚(:Ç  ŠŠ‹Óÿʇ´ disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2921.27734375( disk./.usagePercent‚70.5140394048987 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0# disk.dm-0.out‚47139.14892578125 disk.drbd1552.out‚0 disk.drbd1289.out‚0 cpu‚25.3366896731213* proc.memory.percent‚11.886707941691082 network.recv‚ 1.333105e+06 disk.drbd2041.out‚0 disk.drbd2111.out‚0$ memory_percent‚92.12632179260254 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚47139.14892578125 disk.sda.out‚ 795.8671875 disk.drbd1673.in‚0 network.sent‚ 1.7915396e+07 disk.drbd2118.in‚0 disk.sda2.out‚ 795.8671875 disk.drbd2041.in‚0 disk.rbd1.out‚ 94.9423828125 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd1510.in‚0 disk.drbd1289.in‚0% disk./.usageGB‚1260.8626403808594 disk.dm-0.in‚ 106.52734375 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1935.03515625 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd1356.in‚0m.¾$4Ú+/ _wandbruntime‚18635  _runtime‚18635ø­›ÎËÛ+Å _step‚852 ' lr-AdamW/pg1‚2.0306051610635404e-05 ' lr-AdamW/pg2‚2.0306051610635404e-05 trainer/global_step‚10249 % _timestamp‚1.7848578724148028e+09 _runtime‚18635.772054761Z <ÐÜ+Ê ' lr-AdamW/pg1‚2.0306051610635404e-05 ' lr-AdamW/pg2‚2.0306051610635404e-05 trainer/global_step‚10249 % _timestamp‚1.7848578724148028e+09 _runtime‚18635.772054761 _step‚852ÔWþ•?4Ý+/ _wandbruntime‚18635  _runtime‚18635Aìý‚Þ+ŠP YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2951092673812480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2949929015796112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 trainer/global_step‚10249 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95169965 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13080833706496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.lñþ@Ž arge_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31227428 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10695566553088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2961789585801728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚807465428605952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88985710 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2951092647639040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88985716 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10679002350254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95180029 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10677657738968 $ _timestamp‚1.784857872415875e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2961788214192128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10696911989248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820546692156416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2960606648107112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10696911989248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95169965 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27312820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚395 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚18635.776931245 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚807465796697088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2961788214192128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88985710 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88985716 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58540248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31227033 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13080895459328 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2949928990368144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820546262312448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95180029 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2951092647639040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10695566553088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2951092673812480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58539848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27312815 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070Úº(ß+P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10696911989248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820546692156416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2960606648107112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10696911989248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95169965 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27312820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚395 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _runtime‚18635.776931245 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚807465796697088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2961788214192128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88985710 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88985716 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58540248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31227033 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13080895459328 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2949928990368144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820546262312448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95180029 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2951092647639040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10695566553088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2951092673812480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58539848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27312815 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2951092673812480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2949929015796112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 trainer/global_step‚10249 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95169965 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13080833706496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31227428 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10695566553088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2961789585801728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚807465428605952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88985710 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2951092647639040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88985716 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10679002350254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95180029 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10677657738968 $ _timestamp‚1.784857872415875e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2961788214192128Õ‹.&¸4à+/ _wandbruntime‚18637  _runtime‚18637£A)O'á+”N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10697934504448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚820626086435328 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2960886470682528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2960887840721894 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88994077 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚387 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31230190 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10680023152006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2962068145990656 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2951371583102976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚10249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10696589061120 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58545263 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 $ _timestamp‚1.784857873893372e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚807544383063040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95189125 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2950207792141920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88994077 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10697934504448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚382 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184173124 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95179047 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31229808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚807544014971904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚820626514174976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95179047 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10678678540608 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184183208 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2962069517607424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184173124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58545650 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13082071463424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27315455 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2962069517607424 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2951371583102976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88994083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2950207817569888 _step‚854 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2951371556929536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27315460 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184183208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528Zâgìâ+™N PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10697934504448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚382 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184173124 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95179047 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31229808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚807544014971904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚820626514174976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95179047 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10678678540608 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184183208 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2962069517607424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184173124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58545650 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13082071463424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27315455 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceSt©·kì3atsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2962069517607424 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2951371583102976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88994083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2950207817569888 _step‚854 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2951371556929536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27315460 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184183208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10697934504448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚820626086435328 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2960886470682528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2960887840721894 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88994077 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚387 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31230190 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10680023152006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2962068145990656 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2951371583102976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚10249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10696589061120 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58545263 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 $ _timestamp‚1.784857873893372e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚807544383063040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95189125 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2950207792141920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88994077 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F4õ+/ _wandbruntime‚18722  _runtime‚18722QðÄ&(ö+‹P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10746756109312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185040047 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2965038112163840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185029977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89404066 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27444815 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2963869104484912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚383 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58818326 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10730106233062 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2975784868273152 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2975784868273152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2965038138337280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚857 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811395114994688 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2974597840678720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10748101545472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚824538039105024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10746756109312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚811394746903552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89404060 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13142872843264 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2975786239882752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58818709 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27444820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚824537619746816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2965038138337280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 _runtime‚18722.731913004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95635981 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31373511 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2974599210717974 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89404060 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10748101545472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 trainer/global_step‚10299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89404066 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31373889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2975786239882752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848579593715277e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89404060 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13142872843264 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2975786239882752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58818709 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27444820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚824537619746816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2965038138337280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 _runtime‚18722.731913004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95635981 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31373511 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2974599210717974 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89404060 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10748101545472 P@DeviceSt¯ÅKÆatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 trainer/global_step‚10299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89404066 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31373889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2975786239882752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 % _timestamp‚1.7848579593715277e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10746756109312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185040047 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2965038112163840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185029977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89404066 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27444815 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2963869104484912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚383 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58818326 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10730106233062 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2975784868273152 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2975784868273152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2965038138337280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚857 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811395114994688 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2974597840678720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10748101545472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚824538039105024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10746756109312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚811394746903552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2965317021454336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚377 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58823964 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2976066171688448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58824341 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811473701360640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚18724.193399801 _step‚858 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89412433 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚824617458845696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95634999 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185047426 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13144178933248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89412433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10749124060672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185057510 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10747778617344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚811473333269504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95634999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31376509 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10729782423416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185057510 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2974877663254136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10749124060672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27447455“ç 'ú+šN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31376881 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2964147906258688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27447460 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2976066171688448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185057510 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10747778617344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚811473333269504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95634999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31376509 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10729782423416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10299 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185057510 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2974877663254136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10749124060672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27447455 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2965317021454336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚377 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58823964 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2976066171688448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58824341 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811473701360640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚18724.193399801 _step‚858 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89412433 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚824617458845696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95634999 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185047426 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13144178933248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89412433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10749124060672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2978983602862080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚10349 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2989781522354176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89822416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96091933 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10779865504584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13205314715648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚815324433292288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚861 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2978983576688640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10797945665536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31522652 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2989781522354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185904279 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2989782893963776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2978983602862080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10799291101696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59099467 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚828529797177856 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2988590403289582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59099837 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31523017 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96081869 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89822410 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚828529379916800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89822416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27576815 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 $ _timestamp‚1.784858045223588e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27576820 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10797945665536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚815324065201152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13205363885568 _runtime‚18808.584926504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2989782893963776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112Ä:ß(,P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185904279 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2989782893963776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2978983602862080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10799291101696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59099467 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚828529797177856 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2988590403289582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59099837 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31523017 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96081869 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89822410 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚828529379916800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89822416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27576815 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 $ _timestamp‚1.784858045223588e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27576820 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10797945665536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚815324065201152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13205363885568 _runtime‚18808.584926504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2989782893963776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2978983602862080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 trainer/global_step‚10349 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2989781522354176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89822416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96091933 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10779865504584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13205314715648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚815324433292288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚861 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2978983576688640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10797945665536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31522652 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2989781522354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ݳâVuNŽ,‚(:à  ¾‹‹ÓŠÂù³ network.sent‚ 1.8065721e+07) disk./.usagePercent‚70.51401017761451 disk.drbd1552.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0# proc.memory.rssMB‚ 2921.27734375 network.recv‚ 1.347023e+06 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.12786356608072 disk.sda2.out‚ 799.9453125 disk.drbd1828.in‚0& proc.memory.availableMB‚ 1934.65625 disk.drbd1552.in‚0 disk.sda.out‚ 799.9453125 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.886707941691082 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.nvme0n1.in‚ 106.52734375 cpu‚25.336127241044146$ disk./.usageGB‚1260.862117767334 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 106.52734375 disk.drbd2078.out‚0" disk.dm-0.out‚47145.5126953125 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 95.669921875% disk.nvme0n1.out‚47145.5126953125cN„mž,‚(:“ ¾‹‹ÓŽ‚ܳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚269.261' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.75366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Η÷4,/ _wandbruntime‚18810  _runtime‚18810^ýhà‘,•N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _step‚862 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59105047 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27579455 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2990062825769472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31525962 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚815403019658240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2978087969519520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch·ì`Á;_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848580467022738e+09 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185921728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚828609219532800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185931812 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13206621322752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2988870225865110 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96090951 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2979262512152576 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2978087994947488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2979262485979136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10800313616896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2979262485979136 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96101029 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59105422 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2979262512152576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89830777 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13206567965696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10798968173568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2979262512152576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89830777 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13206567965696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10798968173568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59105047 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27579455 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2990062825769472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31525962 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚815403019658240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2978087969519520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848580467022738e+09 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185921728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚828609219532800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185931812 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13206621322752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2988870225865110 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96090951 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2979262512152576 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2978087994947488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2979262485979136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10800313616896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2979262485979136 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÞdÜ¥r4“,/ _wandbruntime‚18810  _runtime‚18810é?ëó ”,‡ 3loss/SRqqt1/assignment_loss‚0.44327264751015716 0loss/FBt2/detection_loss‚0.43059940487728454 %loss/total_loss‚7.793553816909818 /loss/FRt1/detection_loss‚0.5211955167562485 .loss/FRt2/detection_loss‚1.365949798528603 0loss/SRqqt1/detection_loss‚0.914846751577376 1loss/SRqqt2/detection_loss‚0.6305082774099912 epoch‚36 _runtime‚18810.092814786 .loss/FRt1/assignment_loss‚1.60212721572429 3loss/SRqqt2/assignment_loss‚0.46529147104384316 3loss/FBt1/assignment_loss‚6.337575723769362e-06 3loss/FBt2/assignment_loss‚0.0032669677324490363 trainer/global_step‚10349 % _timestamp‚1.7848580467356017e+09 _step‚863 0loss/FRt2/assignment_loss‚0.6272174039552378 /loss/FBt1/detection_loss‚0.7892720242186139•0î'•,Œ 0loss/SRqqt1/detection_loss‚0.914846751577376 1loss/SRqqt2/detection_loss‚0.6305082774099912 epoch‚36 _runtime‚18810.092814786 .loss/FRt1/assignment_loss‚1.60212721572429 3loss/SRqqt2/assignment_loss‚0.46529147104384316 3loss/FBt1/assignment_loss‚6.337575723769362e-06 3loss/FBt2/assignment_loss‚0.0032669677324490363 trainer/global_step‚10349 % _timestamp‚1.7848580467356017e+09 _step‚863 0loss/FRt2/assignment_loss‚0.6272174039552378 /loss/FBt1/detection_loss‚0.7892720242186139 3loss/SRqqt1/assignment_loss‚0.44327264751015716 0loss/FBt2/detection_loss‚0.43059940487728454 %loss/total_loss‚7.793553816909818 /loss/FRt1/detection_loss‚0.5211955167562485 .loss/FRt2/detection_loss‚1.365949798528603ßç³jΞ–,‚(:“ Í‹‹ÓÏ£³³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚273.517' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.17233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÈŸ‡O—,‚(:Ä  Í‹‹Óã¸Õ³ disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51400847091178 disk.nvme0n1.in‚ 106.52734375 disk.drbd2111.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.886707941691082 disk.drbd2041.in‚0 network.recv‚ 1.348745e+06 disk.sda.out‚ 800.453125 disk.rbd1.out‚ 95.7021484375) proc.memory.availableMB‚ 1934.66796875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0" disk.dm-0.out‚47145.8427734375 disk.drbd1285.out‚0 network.sent‚ 1.8114985e+07 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0% disk./.usageGB‚1260.8620872497559 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0# memory_percent‚92.1278158823649 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 800.453125 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2921.27734375 cpu‚25.33609929605067 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0% disk.nvme0n1.out‚47145.8427734375 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 106.52734375 disk.drbd2041.out‚0%ô¥&R˜,‚(:Ç  Ü‹‹Ó„“Ó³ disk.drbd1285.in‚0 disk.drbd1552.out‚0" proc.memory.rssMB‚ 2925.2265625 disk.sda1.out‚0 network.recv‚ 1.350515e+06 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0% disk./.usageGB‚1260.8595542907715 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0" disk.dm-0.out‚47146.0888671875% disk.nvme0n1.out‚47146.0888671875 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51386681458546 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.sda.out‚ 800.7890625 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 95.8818359375 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.902777353922527 cpu‚25.337673424580785 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.14409192403158 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625) proc.memory.availableMB‚ 1930.66796875 network.sent‚ 1.8165275e+07 disk.sda2.out‚ 800.7890625 disk.drbd1931.out‚0 disk.dm-0.in‚ 106.52734375 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0óEOv™,‚(:’ Ü‹‹Óãúµ³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚114.56' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.18666666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\yv4š,/ _wandbruntime‚18846  _runtime‚18846'œÌŒõ#›,‘v 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3516867160797119 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7735854387283325 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36203083395957947 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5177739262580872 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8694573640823364 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 5 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17982217669487 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6992867588996887 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *jet/accuracy_2_of_3‚0.6090806126594543 epoch‚36 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8198051452636719 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38843870162963867 particle/accuracy_1_of_2‚1 )loss/val_total_loss‚7.766380894463447 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43808525800704956 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7117152214050293 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6473963856697083 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8297850489616394 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07395867258310318 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 trainer/global_step‚10359 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 particle/accuracy_2_of_3‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02796110138297081 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5279157757759094 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 +jet/accuracy_1_of_1‚0.36986103653907776 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29259952902793884 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6248375773429871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7408393025398254 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5971704125404358 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2784333825111389 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017070161178708076 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6566049456596375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17592047154903412 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 +particle/sensitivity‚0.6437097191810608 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7982503175735474 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8095964789390564 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +jet/accuracy_3_of_3‚0.07448111474514008 (particle/accuracy‚0.8555501103401184 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2883544862270355 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _runtime‚18846.98368462 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4523855149745941 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5668178200721741 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 )jet/accuracy_1_of_3‚0.940992534160614 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36489245295524597 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.71656334400177 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3509412109851837 _step‚864 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6682643294334412 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6245883703231812 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29566052556037903 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3509412109851837 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043793462216854095 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03263417258858681 +jet/accuracy_3_of_4‚0.10971427708864212 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5877121686935425 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.386389821767807 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 % _timestamp‚1.7848580836256053e+09 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6473963856697083 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6516643762588501 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 *jet/accuracy_1_of_4‚0.9794445633888245 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9208253622055054 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6914054751396179 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2757920026779175 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8345959782600403 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8352271914482117 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8073673248291016 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 /particle/accuracy_4_of_4‚0.9767659902572632 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6789403557777405 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026383761316537857 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.385256290435791 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7236757278442383 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7162099480628967 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30476102232933044 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8016729354858398 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1487017720937729 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6553049683570862 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2694123685359955 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017070161178708076 particle/accuracy_2_of_2‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6428381204605103 9"Purity/1FRt<ñW†"*SRqqt0FBt/SRqqt_purity‚0.7254542708396912 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09014630317687988 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7501372694969177 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22883903980255127 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5372206568717957 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7248868346214294 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3530089557170868 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9402320981025696 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19141466915607452 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 )jet/accuracy_1_of_2‚0.795624315738678 *jet/accuracy_2_of_2‚0.1716872751712799 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.35931655764579773 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6407684087753296 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8100252151489258 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7317858338356018 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38471412658691406 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.632779061794281 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4008733332157135 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6357056498527527 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4575290083885193 *jet/accuracy_2_of_4‚0.8400989770889282 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6950997114181519 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.30685704946517944 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36271771788597107 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05215795710682869 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7788063287734985 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034759245812892914 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8824449181556702 particle/accuracy_2_of_4‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2662191390991211 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6962758898735046 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2635055482387543 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7616060376167297 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3425498902797699 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7260147929191589 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8824449181556702 particle/accuracy_3_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6603398323059082 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6994566917419434 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6706900000572205 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8285793662071228 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16415010392665863 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 +particle/specificity‚0.9134162068367004 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22527746856212616 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17938895523548126 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7validation_average_jet_accuracy‚0.40505266189575195 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8698833584785461 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26804202795028687 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969.WÞ;œ,–v 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26804202795028687 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3516867160797119 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7735854387283325 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.018756305798888206 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36203083395957947 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5177739262580872 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8694573640823364 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 5 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17982217669487 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6992867588996887 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *jet/accuracy_2_of_3‚0.6090806126594543 epoch‚36 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8198051452636719 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38843870162963867 particle/accuracy_1_of_2‚1 )loss/val_total_loss‚7.766380894463447 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43808525800704956 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7117152214050293 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6473963856697083 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8297850489616394 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07395867258310318 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 trainer/global_step‚10359 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 particle/accuracy_2_of_3‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02796110138297081 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5279157757759094 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 +jet/accuracy_1_of_1‚0.36986103653907776 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29259952902793884 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6248375773429871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7408393025398254 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5971704125404358 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2784333825111389 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017070161178708076 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6566049456596375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17592047154903412 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 +particle/sensitivity‚0.6437097191810608 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7982503175735474 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8095964789390564 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +jet/accuracy_3_of_3‚0.07448111474514008 (particle/accuracy‚0.8555501103401184 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2883544862270355 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _runtime‚18846.98368462 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4523855149745941 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5668178200721741 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 )jet/accuracy_1_of_3‚0.940992534160614 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36489245295524597 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.71656334400177 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3509412109851837 _step‚864 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6682643294334412 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6245883703231812 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29566052556037903 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3509412109851837 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043793462216854095 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03263417258858681 +jet/accuracy_3_of_4‚0.10971427708864212 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5877121686935425 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.386389821767807 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 % _timestamp‚1.7848580836256053e+09 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6473963856697083 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6516643762588501 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 *jet/accuracy_1_of_4‚0.9794445633888245 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9208253622055054 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6914054751396179 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2757920026779175 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8345959782600403 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8352271914482117 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8073673248291016 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 /particle/accuracy_4_of_4‚0.9767659902572632 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6789403557777405 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026383761316537857 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.385256290435791 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7236757278442383 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7162099480628967 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30476102232933044 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8016729354858398 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1487017720937729 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6553049683570862 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2694123685359955 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017070161178708076 particle/accuracy_2_of_2‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6428381204605103 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7254542708396912 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09014630317687988 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7501372694969177 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22883903980255127 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5372206568717957 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7248868346214294 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3530089557170868 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9402320981025696 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19141466915607452 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 )jet/accuracy_1_of_2‚0.795624315738678 *jet/accuracy_2_of_2‚0.1716872751712799 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.35931655764579773 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6407684087753296 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8100252151489258 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7317858338356018 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38471412658691406 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.632779061794281 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4008733332157135 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6357056498527527 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4575290083885193 *jet/accuracy_2_of_4‚0.8400989770889282 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6950997114181519 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.30685704946517944 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36271771788597107 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05215795710682869 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7788063287734985 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034759245812892914 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8824449181556702 particle/accuracy_2_of_4‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2662191390991211 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6962758898735046 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2635055482387543 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7616060376167297 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3425498902797699 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7260147929191589 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8824449181556702 particle/accuracy_3_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6603398323059082 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6994566917419434 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6706900000572205 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8285793662071228 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16415010392665863 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 +particle/specificity‚0.9134162068367004 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22527746856212616 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17938895523548126 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7validation_average_jet_accuracy‚0.40505266189575195 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8698833584785461 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096às_ rL,‚(:Á  ë‹‹Óû„´ disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2927.8671875 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1828.out‚0$ memory_percent‚88.14172744750977 disk.dm-0.in‚ 106.52734375! disk.dm-0.out‚48116.357421875 disk.drbd1285.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 96.0380859375 disk.nvme0n1.in‚ 106.52734375 disk.sda2.out‚ 801.1640625 disk.drbd1285.out‚0* proc.memory.percent‚11.913522084554035 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.sda.out‚ 801.1640625$ disk.nvme0n1.out‚48116.357421875 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51421412859035 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 network.sent‚ 1.8167734e+07 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0( proc.memory.availableMB‚ 2914.2890625# disk./.usageGB‚1260.86576461792 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 network.recv‚ 1.351362e+06 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 cpu‚25.33793358596129›÷ˆ²ž,‚(:’ ë‹‹Óå‚ܳ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚293.18' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.72666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OÊŸŸ,‚(:” ú‹‹Ó³ gpu.0.gpu‚45.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚309.277' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.09233333333331 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/`^2O ,‚(:Ä  ú‹‹ÓÄ¥Ù³ disk.sda2.out‚ 801.55859375 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚48118.4619140625 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd1510.out‚0 disk.rbd1.in‚0$ memory_percent‚88.14217249552408 network.recv‚ 1.353142e+06 disk.dm-0.in‚ 106.52734375" disk.dm-0.out‚48118.4619140625 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 cpu‚25.33790444989777# disk./.usageGB‚1260.86576461792 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.51421412859035 disk.drbd1552.out‚0 disk.sda.out‚ 801.55859375 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2927.8671875 network.sent‚ 1.822185e+07 disk.drbd1931.in‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 2914.1796875 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.rbd1.out‚ 96.0380859375 disk.drbd1226.in‚0* proc.memory.percent‚11.913522084554035 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0…&Ùž¡,‚(:“ ‰Œ‹ÓÞù±³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚283.003' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.33433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒDç’N¢,‚(:à  ‰Œ‹ÓœÎÔ³ cpu‚25.33783485508483 disk.drbd1817.out‚0# disk.dm-0.out‚49089.58349609375 disk.drbd2041.in‚0 disk.sda.in‚ 0.03515625 disk.nvme0n1.in‚ 106.52734375 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 106.52734375 disk.drbd1285.out‚0 disk.drbd1965.out‚0$ memory_percent‚88.14218839009604 proc.cpu.threads‚17% disk./.usageGB‚1260.8639297485352 disk.drbd1550.out‚0 disk.drbd1289.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.sda.out‚ 802.1484375 disk.drbd2111.out‚0 disk.rbd1.out‚ 96.0703125 disk.drbd1285.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0* proc.memory.percent‚11.913522084554035& disk.nvme0n1.out‚49089.58349609375 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 2914.17578125 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2927.8671875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 network.sent‚ 1.822539e+07 disk.drbd1584.out‚0 disk.sda2.out‚ 802.1484375 disk.drbd1289.out‚0 network.recv‚ 1.353971e+06( disk./.usagePercent‚70.5141115130889 disk.drbd2118.in‚0 disk.drbd2111.in‚0x QP£,‚(:Å  ˜Œ‹ÓÛÐé³ disk.drbd2118.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1817.out‚0 network.sent‚ 1.8228432e+07 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 106.52734375 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 802.53515625 disk.sda2.out‚ 802.53515625& disk.nvme0n1.out‚49089.90380859375 disk.sda1.in‚0 network.recv‚ 1.354968e+06 disk.rbd1.out‚96.15625 disk.drbd1510.in‚0 disk.drbd2041.out‚0$ memory_percent‚88.14218839009604) disk./.usagePercent‚70.51406649880448$ disk./.usageGB‚1260.863124847412) proc.memory.availableMB‚ 2914.17578125 cpu‚25.337778010237574 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0" proc.memory.rssMB‚ 2927.8671875 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚49089.90380859375 disk.drbd1285.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.913522084554035 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 106.52734375 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0‹Væឤ,‚(:“ ˜Œ‹ÓȃüÏ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.409' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.13633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[º_ž¥,‚(:“ §Œ‹Ó—¯µ³ gpu.0.gpu‚67.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.16' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.72000000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óŠéìA¦,‚(:¶  §Œ‹Ó¤¾×³$ disk./.usageGB‚1260.863124847412& disk.nvme0n1.out‚49090.73583984375 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 107.921875 disk.drbd1817.in‚0 disk.sda.out‚ 802.96875$ memory_percent‚88.14118703206381 disk.nvme0n1.in‚ 107.921875 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51406649880448 disk.drbd1552.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.sda2.out‚ 802.96875# disk.dm-0.out‚49090.73583984375 disk.drbd1356.out‚0 network.sent‚ 1.8230893e+07 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚96.3125' proc.memory.availableMB‚ 2914.421875 network.recv‚ 1.35588e+06 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2927.8671875 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.913522084554035 cpu‚25.33773531574283ÇbDZ4§,/ _wandbruntime‚18920  _runtime‚18920õ‡ÛȨ, & lr-AdamW/pg2‚1.880595404934504e-05 trainer/global_step‚10399 $ _timestamp‚1.784858156723049e+09 _runtime‚18920.080283354 _step‚865 & lr-AdamW/pg1‚1.880595404934504e-05dOÅ?Í©,Ç & lr-AdamW/pg1‚1.880595404934504e-05 & lr-AdamW/pg2‚1.880595404934504e-05 trainer/global_step‚10399 $ _timestamp‚1.784858156723049e+09 _runtime‚18920.080283354 _step‚865áGÐ!é4ª,/ _wandbruntime‚18920  _runtime‚18920–òK(¾«,ŠP SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3006344471599104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10854753660928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96595197 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13275207980544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_poolàXR$.allocated‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 _runtime‚18920.08161984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2995489744111616 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2995489717938176 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3006344471599104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90330198 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90330192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3005146205851672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90330198 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 % _timestamp‚1.7848581567240884e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96585133 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3006345843208704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10856099097088 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186915325 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27715707 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10856099097088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31697975 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚832577531336704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2995489744111616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186925395 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚10399 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2994308257585712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10854753660928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31698323 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚832577959083520 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10836578226706 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚819302383011840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2995489744111616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186925395 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚10399 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2994308257585712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10854753660928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31698323 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚832577959083520 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10836578226706 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚819302383011840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚348 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3006344471599104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90330198 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90330192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3005146205851672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90330198 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976âu³×4­,/ _wandbruntime‚18921  _runtime‚18921î¢X<'®,”N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90338565 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186942858 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96604293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚832657367982080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚819381337468928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96604293 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚10399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2995768627228672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186932774 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13276398604288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2994587084787456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186932774 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2995768627228672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96594215 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59419236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚338 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 % _timestamp‚1.7848581582018833e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31700889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10857121612288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚832657787333120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3005424658387834 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚819380969377792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10855776168960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10855776168960 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2995768653402112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59419579 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186942858 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31701227 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3005426028427200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90338559 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2994587059359488 _runtime‚18921.55950033 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96594215 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27718352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824t¼†( ¯,™N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2994587059359488 _runtime‚18921.55950033 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96594215 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27718352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90338565 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186942858 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96604293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚832657367982080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚819381337468928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96604293 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚10399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2995768627228672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186932774 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13276398604288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2994587084787456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186932774 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2995768627228672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatr™÷sMonitor.on_train_batch_end/allocation.small_pool.freed‚96594215 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59419236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚338 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 % _timestamp‚1.7848581582018833e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31700889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10857121612288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚832657787333120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3005424658387834 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚819380969377792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10855776168960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10855776168960 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2995768653402112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59419579 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186942858 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31701227 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3005426028427200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90338559 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚343ãF¼4°,/ _wandbruntime‚18921  _runtime‚18921žïØ ±,‡ /loss/FRt1/assignment_loss‚1.621418399229884 2loss/SRqqt2/assignment_loss‚0.5376705729119038 _runtime‚18921.596882712 epoch‚37 0loss/FRt2/assignment_loss‚0.6588658201723493 4loss/FBt1/assignment_loss‚1.3323365668243775e-07 /loss/FRt2/detection_loss‚1.5105712062525143 /loss/FBt1/detection_loss‚0.4043878578471349 trainer/global_step‚10399 % _timestamp‚1.7848581582396505e+09 2loss/SRqqt1/assignment_loss‚0.4986260931268515 2loss/FBt2/assignment_loss‚0.007146114796009129 /loss/FRt1/detection_loss‚0.4817917212216068 _step‚868 1loss/SRqqt1/detection_loss‚0.7443058945774025 1loss/SRqqt2/detection_loss‚0.5248007355060955 /loss/FBt2/detection_loss‚0.3996936975694325 %loss/total_loss‚7.389278246444841ä-¡>²,Œ /loss/FBt1/detection_loss‚0.4043878578471349 trainer/global_step‚10399 % _timestamp‚1.7848581582396505e+09 2loss/SRqqt1/assignment_loss‚0.4986260931268515 2loss/FBt2/assignment_loss‚0.007146114796009129 /loss/FRt1/detection_loss‚0.4817917212216068 _step‚868 1loss/SRqqt1/detection_loss‚0.7443058945774025 1loss/SRqqt2/detection_loss‚0.5248007355060955 /loss/FBt2/detection_loss‚0.3996936975694325 %loss/total_loss‚7.389278246444841 /loss/FRt1/assignment_loss‚1.621418399229884 2loss/SRqqt2/assignment_loss‚0.5376705729119038 _runtime‚18921.596882712 epoch‚37 0loss/FRt2/assignment_loss‚0.6588658201723493 4loss/FBt1/assignment_loss‚1.3323365668243775e-07 /loss/FRt2/detection_loss‚1.5105712062525143äqR‹YG³,‚(:¼  ¶Œ‹ÓÏëî³ disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1226.out‚0 cpu‚25.3376653961256) disk./.usagePercent‚70.51406649880448 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚49090.92724609375 disk.drbd1510.in‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0$ disk./.usageGB‚1260.863124847412 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 network.recv‚ 1.356798e+06 disk.rbd1.out‚96.3125 disk.drbd1931.in‚0 disk.dm-0.in‚ 107.921875 disk.sda1.out‚0$ memory_percent‚88.14291954040527 proc.cpu.threads‚17) proc.memory.availableMB‚ 2913.99609375 network.sent‚ 1.8234475e+07 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.sda.out‚ 803.2421875 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2927.8671875 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 803.2421875# disk.dm-0.out‚49090.92724609375 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.nvme0n1.in‚ 107.921875 disk.drbd1550.in‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.913522084554035 disk.drbd2041.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0ö¢å´,‚(:’ ¶Œ‹ÓȞ̳ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚276.91' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.30333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š£µ,‚(:’ ÅŒ‹Ó÷ˆÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚213.31' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.10333333333332 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 HwN¶,‚(:à  ÅŒ‹ÓêŒø³ disk.drbd2078.out‚0 disk.rbd1.out‚ 96.3447265625 disk.drbd1938.in‚0 disk.dm-0.in‚ 107.921875 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.913522084554035 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 107.921875 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625" proc.memory.rssMB‚ 2927.8671875$ memory_percent‚88.14295132954916 disk.drbd2041.out‚0 network.sent‚ 1.8293906e+07 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 cpu‚25.33763542718925 disk.rbd1.in‚0& disk.nvme0n1.out‚49091.34228515625 disk.drbd1552.in‚0 disk.drbd2118.out‚0) disk./.usagePercent‚70.51406649880448$ disk./.usageGB‚1260.863124847412# disk.dm-0.out‚49091.34228515625 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0) proc.memory.availableMB‚ 2913.98828125 network.recv‚ 1.358325e+06 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 803.6328125 disk.sda2.out‚ 803.6328125á(”È‘·,‚(:† ÔŒ‹Ó»œÚ³ gpu.0.gpu‚43.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.06' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.02 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G{BQ¸,‚(:Æ  ÔŒ‹ÓÜéø³ disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 96.3447265625 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 network.sent‚ 1.8297002e+07 disk.drbd1584.in‚0 disk.nvme0n1.in‚ 107.921875 disk.drbd1931.out‚0 disk.dm-0.in‚ 107.921875 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚ 0.03515625* proc.memory.percent‚11.913522084554035$ memory_percent‚88.14299901326498 disk.drbd1552.in‚0 disk.drbd2118.in‚0 cpu‚25.337592523366222 proc.cpu.threads‚17 disk.sda.out‚ 803.96484375 disk.drbd2111.in‚0 disk.drbd1285.in‚0 network.recv‚ 1.359213e+06 disk.drbd2078.in‚0& disk.nvme0n1.out‚49091.57275390625 disk.drbd1510.out‚0( proc.memory.availableMB‚ 2913.9765625) disk./.usagePercent‚70.51406820550721 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625# disk.dm-0.out‚49091.57275390625 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8631553649902 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 803.96484375 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2927.8671875Ówñ'ž¹,‚(:“ ㌋ÓÞýâ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.455' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.81833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚¬HèQº,‚(:Æ  ㌋Ó£÷…´ disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 96.5009765625 disk.drbd1510.in‚0 disk.drbd2118.out‚0# disk.dm-0.out‚49091.96630859375 disk.sda2.out‚ 804.23828125 disk.rbd1.in‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0" proc.memory.rssMB‚ 2927.8671875 disk.dm-0.in‚ 107.921875% disk./.usageGB‚1260.8631591796875 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 cpu‚25.337523119069157 disk.drbd1510.out‚0 disk.sda.out‚ 804.23828125 proc.cpu.threads‚17 network.recv‚ 1.360155e+06 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.sda2.in‚ 0.03515625 network.sent‚ 1.8300042e+07 disk.drbd1356.in‚0& disk.nvme0n1.out‚49091.96630859375 disk.drbd1226.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1938.in‚0$ memory_percent‚88.14299901326498 disk.drbd1356.out‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.913522084554035 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 2913.9765625 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 107.921875 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.51406841884506 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0°ƒAž»,‚(:“ òŒ‹ÓÚÕ±³ gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚267.778' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.25933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jP,M¼,‚(:  òŒ‹Ó­ˆÒ³ disk.sda.out‚ 804.671875 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.out‚0 disk.sda2.out‚ 804.671875 disk.drbd1356.out‚0 disk.drbd1552.in‚0 network.sent‚ 1.8303035e+07 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 107.921875* proc.memory.percent‚11.913522084554035 network.recv‚ 1.361197e+06 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0$ memory_percent‚88.13859621683756 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0 cpu‚25.33746838079905 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.51406841884506 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚49092.18115234375 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 107.921875 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8631591796875 disk.drbd1673.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚49092.18115234375" proc.memory.rssMB‚ 2927.8671875 disk.rbd1.out‚ 96.5009765625 disk.drbd1550.out‚0) proc.memory.availableMB‚ 2915.05859375ÆÚ¥O½,‚(:Ä  ‹Ó‰‹Ô³ disk.drbd1285.out‚0& disk.nvme0n1.out‚49092.50244140625 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0$ memory_percent‚88.13761075337729# disk.dm-0.out‚49092.50244140625 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 network.recv‚ 1.36201e+06 disk.drbd1289.out‚0) disk./.usagePercent‚70.51406841884506 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 805.05078125 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.913522084554035 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625) proc.memory.availableMB‚ 2915.30078125 disk.dm-0.in‚ 107.921875 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2041.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 96.533203125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 805.05078125 disk.drbd1931.out‚0 proc.cpu.threads‚17 network.sent‚ 1.830658e+07 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0" proc.memory.rssMB‚ 2927.8671875 disk.nvme0n1.in‚ 107.921875 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 cpu‚25.337438823406313% disk./.usageGB‚1260.8631591796875 disk.drbd2041.in‚03™d­œ¾,‚(:‘ ‹Ó ú®³ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.869' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.95633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/{H4¿,/ _wandbruntime‚19006  _runtime‚19006ï¹ÁõÊÀ,Ä ' lr-AdamW/pg1‚1.8316257681225006e-05 ' lr-AdamW/pg2‚1.8316257681225006e-05 trainer/global_step‚10449 $ _timestamp‚1.784858243196423e+09 _runtime‚19006.553647798 _step‚869÷³Ê ÏÁ,É ' lr-AdamW/pg1‚1.8316257681225006e-05 ' lr-AdamW/pg2‚1.8316257681225006e-05 trainer/global_step‚10449 $ _timestamp‚1.784858243196423e+09 _runtime‚19006.553647798 _step‚869åNÌáÜ4Â,/ _wandbruntime‚19006  _runtime‚19006~æ³(Ã,‹P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90748548 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3020342497289728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31845522 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚836569766616064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187789627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3009435182462976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3009435182462976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90748542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3019136028384026 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚10449 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90748542 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187799697 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90748548 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97041085 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚823232069400576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚19006.557930271 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3008248346274512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3009435208636416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31845161 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13337645948416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27847712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3008248371702480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10887682109514 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97041085 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3020342497289728 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚836569347257856 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3009435208636416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13337697215488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10889026720800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97051149 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3019137398423280 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7848582431975005e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3020341125680128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10907288653312 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59692868 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10905943217152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97051149Æe‚%”Ä,P GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3019136028384026 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚10449 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90748542 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187799697 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90748548 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97041085 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚823232069400576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚19006.557930271 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3008248346274512 aLDeviceStatsMonitor.on_train_batch_start/allocated_byõ¶cu‚#tes.large_pool.allocated‚3009435208636416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31845161 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13337645948416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27847712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3008248371702480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10887682109514 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97041085 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3020342497289728 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚836569347257856 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3009435208636416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13337697215488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10889026720800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97051149 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3019137398423280 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7848582431975005e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3020341125680128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10907288653312 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59692868 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10905943217152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97051149 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90748548 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3020342497289728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31845522 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚836569766616064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187789627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3009435182462976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3009435182462976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90748542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005æÃ=½ó4Å,/ _wandbruntime‚19008  _runtime‚19008U\¬5'Æ,”N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚19008.02810906 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90756909 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3009714091753472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚836649613186048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3020621057478656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10908311168512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10906965725184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97060245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10906965725184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97050167 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3019415850959442 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187817160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823310287675392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3020622429095424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚823310655766528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13338957419520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13338908256768 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3008527173476256 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90756915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10449 _step‚871 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97050167 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3009714091753472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187807076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10890047522552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3009714117926912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10908311168512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 % _timestamp‚1.7848582446705399e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59698426 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3009714117926912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187817160 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90756909 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10888702911154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187807076 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27850347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31848079 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3019417220998808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97060245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3020622429095424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31848430 Ù^i'Ç,™N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90756915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10449 _step‚871 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97050167 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3009714091753472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187807076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10890047522552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3009714117926912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10908311168512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 % _timestamp‚1.7848582446705399e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59698426 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3009714117926912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187817160 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90756909 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10888702911154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187807076 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27850347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31848079 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3019417220998808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97060245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3020622429095424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31848430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚19008.02810906 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90756909 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3009714091753472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚836649613186048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3020621057478656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10908311168512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10906965725184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97060245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10906965725184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97050167 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚356 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3019415850959442 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187817160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823310287675392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3020622429095424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚823310655766528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13338957419520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13338908256768 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3008527173476256 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3034337779761152 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3023380673161216 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3023380646987776 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13400378112000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10958478209536 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31992566 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97507101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚827161387698176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10499 % _timestamp‚1.7848583299854772e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3023380646987776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91166898 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59972273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13400326844928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188673999 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10938785992322 _runtime‚19093.343053459 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3034339151370752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91166892 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91166892 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3022188434963312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10940130603608 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10958478209536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3022188460391280 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13400378112000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10958478209536 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31992566 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97507101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚827161387698176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10499 % _timestamp‚1.7848583299854772e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3023380646987776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91166898 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59972273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13400326844928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188673999 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10938785992322 _runtime‚19093.343053459 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3034339151370752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91166892 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91166892 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3022188434963312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10940130603608 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10958478209536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3022188460391280 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3034337779761152 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3023380673161216 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3023380646987776 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59978170 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3033408413570416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 $ _timestamp‚1.784858331462843e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3034619083176448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚840641609724928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3022467236737088 trainer/global_step‚10499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91175265 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10959500724736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10939806793962 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188691462 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188681378 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 _runtime‚19094.820433366 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91175259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59977809 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3034617711559680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3023659582451712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91175265 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚827239605972992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3034619083176448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31995818 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188691462 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10958155281408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27982347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31995462 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚875 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3023659556278272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13401635660800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97506119 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13401582303744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3022467262165056 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10958155281408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3023659556278272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97506119 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188681378 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91175259 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10959500724736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97516197ÖÓ«t'Ý,™N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188691462 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188681378 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 _runtime‚19094.820433366 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91175259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59977809 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3034617711559680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3023659582451712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91175265 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚827239605972992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3034619083176448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31995818 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188691462 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10958155281408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27982347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31995462 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _step‚875 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3023659556278272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13401635660800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97506119 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13401582303744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚356 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3022467262165056 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10958155281408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3023659556278272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97506119 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚361 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188681378 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91175259 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10959500724736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97516197 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27982352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10941151405360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚827239974064128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59978170 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3033408413570416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 $ _timestamp‚1.784858331462843e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3034619083176448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚840641609724928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3022467236737088 trainer/global_step‚10499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91175265 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10959500724736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10939806793962ë~-Å4Þ,/ _wandbruntime‚19094  _runtime‚19094ÀBå ß,‡ 0loss/FRt1/assignment_loss‚1.5373425311701638 /loss/FRt2/detection_loss‚1.3745302958952825 _step‚876 1loss/FBt2/assignment_loss‚0.03522158759643634 0loss/SRqqt1/detection_loss‚0.963576690600158 1loss/SRqqt2/detection_loss‚0.5718866619331061 0loss/FBt1/detection_loss‚0.37500016641182204 /loss/FBt2/detection_loss‚0.3076464072773888 % _timestamp‚1.7848583314998233e+09 0loss/FRt2/assignment_loss‚0.6112868278307686 3loss/SRqqt2/assignment_loss‚0.49301721504735985 .loss/FRt1/detection_loss‚0.521278495108189 epoch‚37 trainer/global_step‚10499 2loss/SRqqt1/assignment_loss‚0.5749649471067521 4loss/FBt1/assignment_loss‚1.3152544075486125e-06 %loss/total_loss‚7.365753141231835 _runtime‚19094.857107646X/±à,Œ %loss/total_loss‚7.365753141231835 _runtime‚19094.857107646 0loss/FRt1/assignment_loss‚1.5373425311701638 /loss/FRt2/detection_loss‚1.3745302958952825 _step‚876 1loss/FBt2/assignment_loss‚0.03522158759643634 0loss/SRqqt1/detection_loss‚0.963576690600158 1loss/SRqqt2/detection_loss‚0.5718866619331061 0loss/FBt1/detection_loss‚0.37500016641182204 /loss/FBt2/detection_loss‚0.3076464072773888 % _timestamp‚1.7848583314998233e+09 0loss/FRt2/assignment_loss‚0.6112868278307686 3loss/SRqqt2/assignment_loss‚0.49301721504735985 .loss/FRt1/detection_loss‚0.521278495108189 epoch‚37 trainer/global_step‚10499 2loss/SRqqt1/assignment_loss‚0.5749649471067521 4loss/FBt1/assignment_loss‚1.3152544075486125e-06ìNgÌ“á,‚(:ˆ Û‹Ó—Üس gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚114.531' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.177 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FoݲNâ,‚(:à  Û‹Ó—ãü³ network.recv‚ 1.368519e+06 disk.drbd1226.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 107.921875 network.sent‚ 1.8381167e+07 disk.drbd1552.in‚0 disk.sda1.in‚0* proc.memory.percent‚11.913522084554035 disk.drbd1828.in‚0 disk.drbd1673.out‚0( proc.memory.availableMB‚ 2914.5859375 disk.rbd1.out‚ 96.8779296875 disk.nvme0n1.in‚ 107.921875 disk.sda.out‚ 806.8984375 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0) disk./.usagePercent‚70.51406841884506 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚49094.57666015625 disk.drbd1226.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 cpu‚25.33714416543599 disk.drbd1584.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 806.8984375 disk.drbd1356.out‚0$ memory_percent‚88.14051946004233 disk.rbd1.in‚0 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8631591796875 disk.drbd1510.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚49094.57666015625 disk.sda1.out‚0 disk.drbd1817.in‚0" proc.memory.rssMB‚ 2927.8671875ÿ—¨hžã,‚(:“ ê‹Óĉ¿³ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.68' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.22666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ßqòLä,‚(:Á  ê‹Ó“ƒß³* proc.memory.percent‚11.913522084554035 network.recv‚ 1.370479e+06) disk./.usagePercent‚70.51406841884506 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625) proc.memory.availableMB‚ 2913.77734375 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0 cpu‚25.3371026564472 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.dm-0.in‚ 107.921875% disk./.usageGB‚1260.8631591796875 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0$ memory_percent‚88.14380963643393 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 807.328125 disk.rbd1.out‚ 96.8779296875 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0" proc.memory.rssMB‚ 2927.8671875 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚49094.74462890625 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 107.921875 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 network.sent‚ 1.8441267e+07& disk.nvme0n1.out‚49094.74462890625 disk.drbd1226.out‚0 disk.sda.out‚ 807.328125 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0ÌÁ~rJå,‚(:¿  ù‹Óî¨ñ³ disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 807.8046875 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1285.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0 network.recv‚ 1.371534e+06 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 807.8046875# disk.dm-0.out‚49095.04833984375* proc.memory.percent‚11.913522084554035 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1226.out‚0$ memory_percent‚84.92418924967447 disk.drbd1285.in‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17 network.sent‚ 1.8444258e+07) disk./.usagePercent‚70.51406841884506 disk.drbd1817.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1289.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 107.921875 disk.rbd1.out‚ 96.91015625& disk.nvme0n1.out‚49095.04833984375 disk.drbd1817.out‚0 disk.dm-0.in‚ 107.921875" proc.memory.rssMB‚ 2927.8671875 cpu‚25.33704710378744% disk./.usageGB‚1260.8631591796875& proc.memory.availableMB‚ 3705.03125ÚÌì=Ÿæ,‚(:” ù‹Ó¯ëѳ gpu.0.gpu‚53.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.011' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.33700000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ªzœŸç,‚(:” ˆŽ‹Óå¡Þ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚135.312' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚45.104000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ ñQLè,‚(:Á  ˆŽ‹Ó¡²‚´% disk./.usageGB‚1260.8631591796875 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625$ memory_percent‚86.08260154724121) proc.memory.availableMB‚ 3420.33984375 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.out‚ 808.078125) disk./.usagePercent‚70.51406841884506 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 network.sent‚ 1.8447356e+07# disk.dm-0.out‚49095.34716796875 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.913522084554035 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 96.91015625 disk.drbd1289.in‚0" proc.memory.rssMB‚ 2927.8671875& disk.nvme0n1.out‚49095.34716796875 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 808.078125 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 107.921875 network.recv‚ 1.372621e+06 disk.dm-0.in‚ 107.921875 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0 cpu‚25.337017754707013 disk.drbd1510.in‚0 disk.drbd1584.in‚0³bóžé,‚(:“ —Ž‹ÓÕÏÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.702' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.23400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mg$Bê,‚(:·  —Ž‹Ó©¯ñ³ disk.sda2.out‚808.3125 disk.drbd1828.in‚0 disk.drbd1226.out‚0! disk.dm-0.out‚49096.134765625 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8631629943848 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 network.recv‚ 1.373629e+06 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0$ memory_percent‚87.27356592814127* proc.memory.percent‚11.913522084554035 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 97.06640625 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2927.8671875 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.sda.out‚808.3125 disk.drbd1965.in‚0 disk.drbd1356.in‚0( proc.memory.availableMB‚ 3127.6484375( disk./.usagePercent‚70.5140686321829 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 107.921875 cpu‚25.336937064653764 network.sent‚ 1.8450395e+07 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 107.921875$ disk.nvme0n1.out‚49096.134765625 proc.cpu.threads‚17y…"¥žë,‚(:“ ¦Ž‹Óæ–Û³ gpu.0.gpu‚25.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚309.189' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.06300000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸLì,‚(:Á  ¦Ž‹ÓÇ÷ú³% disk./.usageGB‚1260.8597869873047 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 108.046875 disk.drbd1550.in‚0 network.sent‚ 1.8453391e+07 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0& proc.memory.availableMB‚ 2817.46875 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0) disk./.usagePercent‚70.51387982819375 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 108.046875 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0$ memory_percent‚88.53569030761719 cpu‚25.33693404295432# disk.dm-0.out‚49096.49560546875 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 97.24609375* proc.memory.percent‚11.913522084554035 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 network.recv‚ 1.374497e+06 disk.drbd2111.in‚0& disk.nvme0n1.out‚49096.49560546875" proc.memory.rssMB‚ 2927.8671875 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.sda.out‚ 808.82421875 disk.sda2.out‚ 808.82421875 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0„&ãg4í,/ _wandbruntime‚19179  _runtime‚19179§=Èî, & lr-AdamW/pg2‚1.735275531328092e-05 trainer/global_step‚10549 % _timestamp‚1.7848584165683508e+09 _runtime‚19179.92685045 _step‚877 & lr-AdamW/pg1‚1.735275531328092e-05+¨Íï,Ç & lr-AdamW/pg1‚1.735275531328092e-05 & lr-AdamW/pg2‚1.735275531328092e-05 trainer/global_step‚10549 % _timestamp‚1.7848584165683508e+09 _runtime‚19179.92685045 _step‚877íÿ•rÁ4ð,/ _wandbruntime‚19179  _runtime‚19179ñ3É|ñ,ŠP H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97952989 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3036128523652112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚10549 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3037326111512576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3037326111512576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3037326137686016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pooléæ›!.freed‚831090337904640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 $ _timestamp‚1.784858416569078e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13463227202048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13463175934976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3036128549080080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3047118413527242 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91585242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32140750 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3048334433842176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11009667765760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚19179.926966516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91585248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚844553933197824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28111712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3048334433842176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3037326137686016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11008322329600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32140404 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11009667765760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28111707 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189538231 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10991234486416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91585242 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60252111 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60252462 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97952989 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91585248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97963053 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97963053 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3048334433842176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3037326137686016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11008322329600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32140404 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11009667765760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28111707 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189538231 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10991234486416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91585242 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60252111 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60252462 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97952989 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91585248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97963053 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97963053 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97952989 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3036128523652112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 trainer/global_step‚10549 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3037326111512576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3037326111512576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3037326137686016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚831090337904640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 $ _timestamp‚1.784858416569078e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13463227202048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13463175934976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3036128549080080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3047118413527242 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91585242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32140750 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3048334433842176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11009667765760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 _runtime‚19179.926966516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91585248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚844553933197824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28111712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1îÀk44ó,/ _wandbruntime‚19181  _runtime‚19181¦Å'ô,•N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3037605046976512 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3048614365640704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28114352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32143671 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚844633352344576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28114347 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32143323 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13464477236736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _step‚879 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97962071 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚844633769598464 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3047399606142024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11010690280960 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189555680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 _runtime‚19181.401900935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91593615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97972149 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3036407325425888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97962071 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3037605020803072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10992255288168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11009344837632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91593609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91593609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13464428073984 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3047398236102658 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3036407350853856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 trainer/global_step‚10549 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60257670 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3037605020803072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11010690280960 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189565764 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚831169292361728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10990910676770 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97972149 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11009344837632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚348 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 _step‚879 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97962071 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚844633769598464 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3047399606142024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11010690280960 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189555680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 _runtime‚19181.401900935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_po š\(ol.allocated‚91593615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97972149 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3036407325425888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97962071 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3037605020803072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10992255288168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11009344837632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91593609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91593609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13464428073984 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3047398236102658 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3036407350853856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 trainer/global_step‚10549 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60257670 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3037605020803072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11010690280960 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189565764 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚831169292361728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10990910676770 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97972149 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11009344837632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚348 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190412533 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3062331087923200 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3062332459532800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3051271602210816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13526047647744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3061110976138104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11042338369224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60531593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28243712 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3061109606098850 trainer/global_step‚10599 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92003592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚339 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11060857321984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13526094720512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3062332459532800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11059511885824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190422603 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3051271602210816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60531937 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3051271576037376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32288225 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚848545703849984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32287886 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92003598 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98419005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚835019656202240 % _timestamp‚1.7848585031792371e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚835020024293376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3062331087923200 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600j?ÎábŠ-P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11059511885824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190422603 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3051271602210816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60531937 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3051271576037376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32288225 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.curre†6?´ nt‚ 415163904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚848545703849984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32287886 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92003598 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98419005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚835019656202240 % _timestamp‚1.7848585031792371e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚835020024293376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3062331087923200 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11059511885824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11040993757938 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190412533 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3062331087923200 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3062332459532800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3051271602210816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13526047647744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3061110976138104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11042338369224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60531593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28243712 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3061109606098850 trainer/global_step‚10599 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92003592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚339 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11060857321984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13526094720512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3062332459532800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824ò+y²Ü4‹-/ _wandbruntime‚19268  _runtime‚19268òÆR'Œ-”N NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92011959 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32291212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848585046564817e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3062611019721728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28246347 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3051550511501312 _step‚883 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚338 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190440066 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚19268.01409731 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190429982 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3051550511501312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3062612391338496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3051550485327872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28246352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10599 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92011965 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60537226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚848625967594496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚835098242568192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60537564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32290879 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13527356935168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3050347439542656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚835098610659328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98428101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11042014559578 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11060534393856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11061879837184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚848625554534912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11061879837184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3062612391338496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11043359170976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13527311966720 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3061389428674266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3051550485327872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190429982 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92011965 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98428101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11060534393856¢:7'-™N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3062612391338496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3051550485327872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28246352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚333 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10599 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92011965 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60537226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚848625967594496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚835098242568192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60537564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32290879 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13527356935168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3050347439542656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚835098610659328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98428101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11042014559578 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11060534393856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11061879837184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚848625554534912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11061879837184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3062612391338496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11043359170976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13527311966720 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3061389428674266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3051550485327872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190429982 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92011965 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98428101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11060534393856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92011959 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32291212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 % _timestamp‚1.7848585046564817e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3062611019721728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28246347 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3051550511501312 _step‚883 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚338 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190440066 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚19268.01409731 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190429982 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3051550511501312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398óvH•ˆ4Ž-/ _wandbruntime‚19268  _runtime‚19268Hï¡-‚ .loss/FBt2/detection_loss‚0.403311433111026 # _timestamp‚1.78485850469081e+09 _runtime‚19268.048016455 0loss/FRt2/assignment_loss‚0.7215974835687847 3loss/SRqqt2/assignment_loss‚0.46885983522565156 1loss/FBt1/assignment_loss‚0.13073006066286447 /loss/FRt1/detection_loss‚0.5587819040641552 1loss/SRqqt1/detection_loss‚0.8662069775633735 trainer/global_step‚10599 _step‚884 /loss/FRt1/assignment_loss‚1.629188815297063 3loss/SRqqt1/assignment_loss‚0.49140466799632987 /loss/FBt1/detection_loss‚0.7916991742271365 %loss/total_loss‚8.246138127812324 epoch‚37 1loss/FBt2/assignment_loss‚0.02324060999131824 1loss/SRqqt2/detection_loss‚0.6097874220489634 /loss/FRt2/detection_loss‚1.5513297440556577›°»P -‡ epoch‚37 1loss/FBt2/assignment_loss‚0.02324060999131824 1loss/SRqqt2/detection_loss‚0.6097874220489634 /loss/FRt2/detection_loss‚1.5513297440556577 .loss/FBt2/detection_loss‚0.403311433111026 # _timestamp‚1.78485850469081e+09 _runtime‚19268.048016455 0loss/FRt2/assignment_loss‚0.7215974835687847 3loss/SRqqt2/assignment_loss‚0.46885983522565156 1loss/FBt1/assignment_loss‚0.13073006066286447 /loss/FRt1/detection_loss‚0.5587819040641552 1loss/SRqqt1/detection_loss‚0.8662069775633735 trainer/global_step‚10599 _step‚884 /loss/FRt1/assignment_loss‚1.629188815297063 3loss/SRqqt1/assignment_loss‚0.49140466799632987 /loss/FBt1/detection_loss‚0.7916991742271365 %loss/total_loss‚8.246138127812324ôÅÛß ž‘-‚(:“ ‹ÓÎÙÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.225' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.74166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï\PN’-‚(:à  ‹ÓÁ¾ü³ disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.913522084554035 disk.nvme0n1.in‚ 108.046875 disk.drbd1226.out‚0 disk.sda1.out‚0 network.sent‚ 1.8532034e+07 disk.drbd2118.out‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.8660888671875 disk.drbd1510.out‚0) disk./.usagePercent‚70.51423226230683 disk.rbd1.out‚ 97.8251953125 disk.drbd1938.out‚0 disk.dm-0.in‚ 108.046875! disk.dm-0.out‚49099.810546875$ memory_percent‚92.09723472595215 disk.drbd1828.in‚0$ disk.nvme0n1.out‚49099.810546875 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.sda.out‚ 811.23046875 disk.drbd1673.in‚0 disk.sda2.out‚ 811.23046875 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1673.out‚0) proc.memory.availableMB‚ 1942.18359375 proc.cpu.threads‚17 disk.sda.in‚ 0.03515625" proc.memory.rssMB‚ 2927.8671875 network.recv‚ 1.382137e+06 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 cpu‚25.336564859284728 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.out‚0]ŠâÇŸ“-‚(:” ž‹Óø¢Ý³ gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.088' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.69600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04£¾”-‚(:Æ  ž‹Óžçü³ disk.sda.out‚ 811.62109375 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1942.18359375 proc.cpu.threads‚17) disk./.usagePercent‚70.514231622293¥ Úä“31% disk./.usageGB‚1260.8660774230957 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 97.8251953125& disk.nvme0n1.out‚49100.02490234375 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.913522084554035 disk.sda1.in‚0 disk.dm-0.in‚ 108.046875 disk.drbd2118.in‚0 network.recv‚ 1.38428e+06 disk.nvme0n1.in‚ 108.046875 disk.drbd1550.in‚0 disk.drbd1817.out‚0 network.sent‚ 1.8591783e+07 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0" proc.memory.rssMB‚ 2927.8671875 disk.drbd1226.out‚0# disk.dm-0.out‚49100.02490234375$ memory_percent‚92.09723472595215 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 811.62109375 disk.drbd1285.out‚0 disk.drbd1965.in‚0 cpu‚25.336523512335162 disk.drbd1356.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0¼ç3*Ÿ•-‚(:” ­‹ÓáÄÞ³ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.879' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.62633333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ÏL–-‚(:Á  ­‹ÓŸý³ disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚49100.6142578125 disk.drbd1584.in‚0 disk.sda.out‚ 811.9609375 disk.drbd1550.in‚0 proc.cpu.threads‚17 network.sent‚ 1.8594852e+07 network.recv‚ 1.385243e+06 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.913522084554035 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0% disk./.usageGB‚1260.8660774230957 disk.sda2.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0" proc.memory.rssMB‚ 2927.8671875 disk.drbd2118.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 108.046875 disk.drbd1550.out‚0" disk.dm-0.out‚49100.6142578125 disk.drbd1226.out‚0$ memory_percent‚92.09736188252768) disk./.usagePercent‚70.51423162229331) proc.memory.availableMB‚ 1942.15234375 disk.drbd1226.in‚0 disk.dm-0.in‚ 108.046875 disk.drbd1817.out‚0 disk.sda2.out‚ 811.9609375 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 97.857421875 disk.drbd1510.in‚0 cpu‚25.33649518769597ã qÔž—-‚(:“ ¼‹Ó¢†Ñ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚285.035' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.01166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U˜ lM˜-‚(:  ¼‹Óõðð³ disk.sda2.in‚ 0.03515625 disk.drbd1552.out‚0) disk./.usagePercent‚70.51423375567171 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd1673.in‚0 network.recv‚ 1.386125e+06 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.sda1.in‚0$ memory_percent‚92.09737777709961( proc.memory.availableMB‚ 1942.1484375 disk.sda.out‚ 812.23046875 disk.sda2.out‚ 812.23046875 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0" disk.dm-0.out‚49100.9462890625 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 97.857421875" proc.memory.rssMB‚ 2927.8671875* proc.memory.percent‚11.913522084554035 disk.drbd1550.out‚0 disk.dm-0.in‚ 108.046875 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 108.046875 cpu‚25.33648007261001 proc.cpu.threads‚17 network.sent‚ 1.8597957e+07 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0% disk.nvme0n1.out‚49100.9462890625 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8661155700684 disk.drbd1817.in‚0 disk.drbd1285.out‚0¥®‚ž™-‚(:“ Ë‹ÓŠþг gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚286.261' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.42033333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Ë5=Hš-‚(:½  Ë‹ÓéÕð³ disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd1938.out‚0" disk.dm-0.out‚49101.2822265625 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 812.53125 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚49101.2822265625 disk.sda2.in‚ 0.03515625 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 812.53125 disk.drbd1289.out‚0) disk./.usagePercent‚70.51423375567171 disk.drbd1584.out‚0" proc.memory.rssMB‚ 2927.8671875 cpu‚25.336426010139274 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.dm-0.in‚ 108.046875% disk./.usageGB‚1260.8661155700684 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 network.sent‚ 1.8600998e+07 disk.nvme0n1.in‚ 108.046875 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1510.out‚0 disk.rbd1.out‚ 98.013671875 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0( proc.memory.availableMB‚ 1942.1484375 network.recv‚ 1.387026e+06 disk.drbd1356.in‚0$ memory_percent‚92.09737777709961 disk.drbd2078.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.913522084554035 disk.drbd1226.in‚0!*œ›-‚(:‘ Ú‹ÓÇþϳ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚59.0 gpu.0.powerWatts‚106.997' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.66566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÈyEœ-‚(:º  Ú‹Ó §ð³# proc.memory.rssMB‚ 2931.81640625 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚49101.5556640625 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1356.out‚0 disk.dm-0.in‚ 108.046875 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 98.013671875 disk.drbd1584.out‚0) proc.memory.percent‚11.92959149678548 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8661155700684 disk.sda.out‚812.9375 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 network.recv‚ 1.388056e+06 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 network.sent‚ 1.8603996e+07 disk.sda2.out‚812.9375 disk.rbd1.in‚0$ memory_percent‚92.11228688557944 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0" disk.dm-0.out‚49101.5556640625 disk.drbd1550.out‚0 disk.drbd2118.out‚0' proc.memory.availableMB‚ 1938.484375 cpu‚25.337984223142406) disk./.usagePercent‚70.51423375567171 disk.nvme0n1.in‚ 108.046875 disk.sda1.out‚0®À¤4-/ _wandbruntime‚19356  _runtime‚193567ï"Ü;ž-˜v ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_1_of_1‚0.3763403594493866 +particle/sensitivity‚0.6472779512405396 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.28893014788627625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2916887402534485 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08893419057130814 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3973383605480194 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5141482949256897 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚37 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746398091316223 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6591838002204895 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9190786480903625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7768883109092712 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8047438859939575 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35097935795783997 particle/accuracy_1_of_1‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7716966867446899 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6941675543785095 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5242987275123596 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7848585933190448e+09 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5813060402870178 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6433259844779968 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39555349946022034 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7152548432350159 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.653377115726471 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6623917818069458 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8724344372749329 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8793770670890808 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1785755455493927 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9733201861381531 5validation_average_jet_accuracy‚0.409686416387558 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28179746866226196 +jet/accuracy_3_of_3‚0.07505069673061371 particle/accuracy_2_of_3‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36198821663856506 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17728270590305328 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3528751730918884 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4541338086128235 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030767839401960373 ,jet/accuracy_4_of_4‚0.018756305798888206 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025838086381554604 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22722360491752625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02423962578177452 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7181400656700134 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8026134371757507 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.663209080696106 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7100723385810852 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2693925201892853 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03293938562273979 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7063719630241394 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8350728750228882 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8410894274711609 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38159266114234924 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6884065866470337 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04265039041638374 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7046493887901306 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6770455241203308 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7346605658531189 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2725207209587097 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3425275385379791 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16621045768260956 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8362786173820496 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8262986540794373 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8417207598686218 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7153812050819397 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22369518876075745 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6717365980148315 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2659878134727478 *jet/accuracy_2_of_2‚0.1708604097366333 *jet/accuracy_2_of_4‚0.8349968791007996 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6674122214317322 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7364124655723572 *jet/accuracy_1_of_3‚0.9417167901992798 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7969301342964172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.660646378993988 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35737255215644836 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07265038788318634 )loss/val_total_loss‚7.752370525039701 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8088290095329285 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6230769753456116 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚10639 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6856521368026733 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4505157172679901 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7347420454025269 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 *jet/accuracy_2_of_3‚0.6031469702720642 particle/accuracy_3_of_4‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7132126688957214 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5361663103103638 +particle/specificity‚0.9135345816612244 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7323066592216492 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 *jet/accuracy_1_of_4‚0.9758731126785278 _step‚885 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7972919344902039 (particle/accuracy‚0.8564104437828064 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_4‚0.10971427708864212 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28608760237693787 _runtime‚19356.677128672 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30120372772216797 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3030553460121155 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40079423785209656 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 particle/accuracy_1_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6674122214317322 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7455937266349792 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9428775906562805 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5987381935119629 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26598668098449707 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7385119199752808 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14731235802173615 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3644505441188812 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36274948716163635 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38951006531715393 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27556130290031433 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7084835171699524 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6232481598854065 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6251630187034607 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051333099603652954 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8083685040473938 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7265251874923706 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8793770670890808 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6087144017219543 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14630743861198425 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3513094484806061 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7648256421089172 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35097935795783997 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4601014256477356 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1599767953157425 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065Ä¡L•*Ÿ-v 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5141482949256897 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚37 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746398091316223 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6591838002204895 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9190786480903625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7768883109092712 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8047438859939575 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35097935795783997 particle/accuracy_1_of_1‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7716966867446899 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6941675543785095 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5242987275123596 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7848585933190448e+09 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5813060402870178 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6433259844779968 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39555349946022034 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7152548432350159 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.653377115726471 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6623917818069458 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8724344372749329 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8793770670890808 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1785755455493927 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9733201861381531 5validation_average_jet_accuracy‚0.409686416387558 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28179746866226196 +jet/accuracy_3_of_3‚0.07505069673061371 particle/accuracy_2_of_3‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36198821663856506 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17728270590305328 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3528751730918884 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4541338086128235 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030767839401960373 ,jet/accuracy_4_of_4‚0.018756305798888206 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025838086381554604 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22722360491752625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02423962578177452 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7181400656700134 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8026134371757507 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.663209080696106 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7100723385810852 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2693925201892853 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03293938562273979 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7063719630241394 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8350728750228882 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8410894274711609 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38159266114234924 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6884065866470337 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04265039041638374 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7046493887901306 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6770455241203308 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7346605658531189 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2725207209587097 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3425275385379791 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16621045768260956 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8362786173820496 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8262986540794373 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8417207598686218 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7153812050819397 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22369518876075745 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6717365980148315 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2659878134727478 *jet/accuracy_2_of_2‚0.1708604097366333 *jet/accuracy_2_of_4‚0.8349968791007996 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.66741222143173P¤_ù22 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7364124655723572 *jet/accuracy_1_of_3‚0.9417167901992798 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7969301342964172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.660646378993988 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35737255215644836 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07265038788318634 )loss/val_total_loss‚7.752370525039701 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8088290095329285 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6230769753456116 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚10639 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6856521368026733 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4505157172679901 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7347420454025269 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 *jet/accuracy_2_of_3‚0.6031469702720642 particle/accuracy_3_of_4‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7132126688957214 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5361663103103638 +particle/specificity‚0.9135345816612244 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7323066592216492 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 *jet/accuracy_1_of_4‚0.9758731126785278 _step‚885 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7972919344902039 (particle/accuracy‚0.8564104437828064 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_4‚0.10971427708864212 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28608760237693787 _runtime‚19356.677128672 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30120372772216797 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3030553460121155 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40079423785209656 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 particle/accuracy_1_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6674122214317322 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7455937266349792 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9428775906562805 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5987381935119629 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26598668098449707 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7385119199752808 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14731235802173615 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3644505441188812 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36274948716163635 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38951006531715393 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27556130290031433 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7084835171699524 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6232481598854065 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6251630187034607 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051333099603652954 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8083685040473938 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7265251874923706 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8793770670890808 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6087144017219543 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14630743861198425 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3513094484806061 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7648256421089172 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35097935795783997 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4601014256477356 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1599767953157425 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_1_of_1‚0.3763403594493866 +particle/sensitivity‚0.6472779512405396 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.28893014788627625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2916887402534485 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08893419057130814 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3973383605480194 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625õ²2gž -‚(:“ é‹Ó®Úϳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚293.879' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.95966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0© ûP¡-‚(:Å  é‹Ó€‰î³ disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 2914.75390625 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 813.32421875 disk.drbd1356.in‚0$ memory_percent‚88.13983599344888 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 cpu‚25.338083770601123 disk.drbd1285.in‚0 disk.drbd2118.out‚0) proc.memory.percent‚11.92959149678548 network.recv‚ 1.389081e+06% disk./.usageGB‚1260.8660583496094 disk.drbd1828.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0& disk.nvme0n1.out‚50167.63037109375 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 813.32421875 disk.drbd1289.in‚0 disk.dm-0.in‚ 108.046875# disk.dm-0.out‚50167.63037109375 disk.rbd1.in‚0 network.sent‚ 1.860699e+07 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0( disk./.usagePercent‚70.5142305556041 disk.rbd1.out‚ 98.0458984375 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 108.046875 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0NM’4¢-/ _wandbruntime‚19377  _runtime‚19377ñY3ŠÉ£-à % _timestamp‚1.7848586144940424e+09 _runtime‚19377.851312079 _step‚886 & lr-AdamW/pg1‚1.641085990539133e-05 & lr-AdamW/pg2‚1.641085990539133e-05 trainer/global_step‚10649y½í€Î¤-È & lr-AdamW/pg2‚1.641085990539133e-05 trainer/global_step‚10649 % _timestamp‚1.7848586144940424e+09 _runtime‚19377.851312079 _step‚886 & lr-AdamW/pg1‚1.641085990539133e-05öt%‰4¥-/ _wandbruntime‚19377  _runtime‚19377UµÅl(¦-‹P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚887 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3078895408777728 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3066567727608848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191433649 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92511374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98922269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3078895408777728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3066567702180880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3078894037168128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11116319881216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3078894037168128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11117665317376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60845943 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3077666778700194 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11097706480060 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3067777743460352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3067777717286912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32463090 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28382604 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191423579 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11099051091346 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92511380 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13595807191552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98922269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13595772701696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚838997974012928 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3077665408660940ý¾w‡(§-P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98922269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13595772701696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚838997974012928 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3077665408660940 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚887 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3078895408777728 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3066567727608848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191433649 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92511374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98922269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3078895408777728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3066567702180880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3078894037168128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11116319881216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3078894037168128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11117665317376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60845943 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3077666778700194 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11097706480060 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3067777743460352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3067777717286912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32463090 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28382604 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191423579 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11099051091346 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92511380 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13595807191552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864÷C74¨-/ _wandbruntime‚19379  _runtime‚19379£'Î@r ©-•N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11118687832576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11100071893098 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3068056652750848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92519747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚839076928470016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11098727281700 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98931365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3077946601275722 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3066846503954656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848586159725642e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚852674059142656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191451112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191441028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28385244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32466129 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11117342389248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚839076560378880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3079175340583424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92519741 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13597130672640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191451112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60851368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98931365 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852673656568832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3077945231236356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98921287 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3079175340583424 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3068056626577408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60851618 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11117342389248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191441028 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3068056626577408 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92519747 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98921287 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32466374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0+©Ä 'ª-šN VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28385244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32466129 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11117342389248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚839076560378880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3079175340583424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92519741 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13597130672640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191451112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60851368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98931365 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852673656568832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3077945231236356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98921287 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3079175340583424 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3068056626577408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60851618 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11117342389248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191441028 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3068056626577408 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92519747 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98921287 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32466374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11118687832576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11100071893098 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3068056652750848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92519747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚839076928470016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11098727281700 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98931365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3077946601275722 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3066846503954656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848586159725642e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚852674059142656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191451112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191441028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34482688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batc~O˜1%h_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11150154974154 % _timestamp‚1.7848587012847445e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32613200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3080507816297648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11168854873600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92929724 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28514599 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61128147 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3081723207985152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚856586587153920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3092890691249152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10699 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192307951 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3081723207985152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92929730 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92929730 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11150154974154 % _timestamp‚1.7848587012847445e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32613200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3080507816297648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11168854873600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92929724 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28514599 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61128147 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3081723207985152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚856586587153920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3092890691249152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10699û‘À"44À-/ _wandbruntime‚19466  _runtime‚19466òuà3'Á-•N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3082002091102208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3091937793847330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92938091 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3082002117275648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848587027633896e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚19466.120987425 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3082002117275648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192325414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99387317 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11149831164508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92938097 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11169877388800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚10699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32616460 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61133704 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3091936423807964 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192315330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92938097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61133362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13660557506560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11168531945472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11151175775906 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99377239 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3080786592643456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28517244 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3093171994664448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192315330 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680/ÏiðG Â-šN _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3080786618071424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚856666436183040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92938091 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3082002117275648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 % _timestamp‚1.7848587027633896e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚19466.120987425 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3082002117275648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192325414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99387317 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11149831164508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92938097 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11169877388800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚10699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32616460 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 N>DeviceStaܪÈÙtsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61133704 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3091936423807964 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192315330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92938097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61133362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13660557506560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11168531945472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚342 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11151175775906 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99377239 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3080786592643456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28517244 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3093171994664448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192315330 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3082002091102208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3091937793847330 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3106887345330176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11220044429824 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3094447904986448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚846856610608128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28646604 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚860578785239552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3094447879558480 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3105649163843410 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3095668646336512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13721806540288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99834173 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3106887345330176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193172183 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61407996 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13721744787456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11201258856962 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3095668672509952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61408361 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93348074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93348080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11220044429824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚846856978699264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11218698993664 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 trainer/global_step‚10749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99824109 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99834173 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193182253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93348074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28646599 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3106888716939776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3095668646336512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11199914245676 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _step‚895 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93348080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚365 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0»wë·µ×-P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28646604 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚860578785239552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3094447879558480 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3105649163843410 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 XIDev{rWöa$iceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3095668646336512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13721806540288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99834173 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3106887345330176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193172183 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61407996 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13721744787456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11201258856962 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3095668672509952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61408361 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93348074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93348080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11220044429824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚846856978699264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11218698993664 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 trainer/global_step‚10749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99824109 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99834173 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193182253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93348074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28646599 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3106888716939776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3095668646336512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11199914245676 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _step‚895 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93348080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚365 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11218698993664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3106887345330176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11220044429824 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3094447904986448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚846856610608128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÿ}O9³4Ø-/ _wandbruntime‚19553  _runtime‚19553çÅ(ú'Ù-•N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11221066945024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99843269 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3094726706760224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3107167277128704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚860658188080128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193199716 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28649244 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93356441 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61413594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32764704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11200935047316 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚10749 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3107168648745472 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32764355 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13722991106048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11202279658714 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚846935565065216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11219721501696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93356447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3105928986418938 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61413948 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193189632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93356447 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3095947555627008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚846935196974080 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193199716 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99843269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11219721501696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚349 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398MDñ 'Ú-šN XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚860658188080128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193199716 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28649244 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93356441 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61413594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32764704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11200935047316 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚354 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚10749 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3107168648745472 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32764355 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13722991106048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11202279658714 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚846935565065216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11219721501696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93356447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3105928986418938 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61413948 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193189632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93356447 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3095947555627008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚846935196974080 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193199716 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99843269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11219721501696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚349 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11221066945024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99843269 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3094726706760224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3107167277128704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0€‰] 64Û-/ _wandbruntime‚19553  _runtime‚19553“ ¸ Ü-‰ _runtime‚19553.100443291 0loss/FRt2/assignment_loss‚0.5932020736063576 /loss/FBt1/detection_loss‚0.6086805555537077 &loss/total_loss‚7.6460776613651795 _step‚897 3loss/FBt1/assignment_loss‚0.0007556305277643917 0loss/FBt2/detection_loss‚0.44568561970129805 epoch‚38 0loss/FRt1/assignment_loss‚1.5888116468503637 2loss/SRqqt1/assignment_loss‚0.5483653155286001 /loss/FRt1/detection_loss‚0.5246359199888564 /loss/FRt2/detection_loss‚1.2952415275736449 0loss/SRqqt1/detection_loss‚0.997999242262067 trainer/global_step‚10749 3loss/SRqqt2/assignment_loss‚0.48564154419442424 2loss/FBt2/assignment_loss‚0.004834727731538461 1loss/SRqqt2/detection_loss‚0.5522238578465576 % _timestamp‚1.7848587897432039e+09¬ÙÎHÝ-Ž /loss/FRt2/detection_loss‚1.2952415275736449 0loss/SRqqt1/detection_loss‚0.997999242262067 trainer/global_step‚10749 3loss/SRqqt2/assignment_loss‚0.48564154419442424 2loss/FBt2/assignment_loss‚0.004834727731538461 1loss/SRqqt2/detection_loss‚0.5522238578465576 % _timestamp‚1.7848587897432039e+09 _runtime‚19553.100443291 0loss/FRt2/assignment_loss‚0.5932020736063576 /loss/FBt1/detection_loss‚0.6086805555537077 &loss/total_loss‚7.6460776613651795 _step‚897 3loss/FBt1/assignment_loss‚0.0007556305277643917 0loss/FBt2/detection_loss‚0.44568561970129805 epoch‚38 0loss/FRt1/assignment_loss‚1.5888116468503637 2loss/SRqqt1/assignment_loss‚0.5483653155286001 /loss/FRt1/detection_loss‚0.5246359199888564¡àòÑKÞ-‚(:À  ¬‘‹Óà­ë³ network.recv‚ 1.403934e+06 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 817.9921875 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0# disk.nvme0n1.out‚51094.87109375 disk.sda.in‚ 0.03515625 cpu‚25.337475146125747 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0) proc.memory.percent‚11.92959149678548 disk.drbd1285.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0( proc.memory.availableMB‚ 3036.1484375# proc.memory.rssMB‚ 2931.81640625 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.dm-0.out‚51094.87109375 disk.drbd1356.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 817.9921875 disk.drbd1673.in‚0$ memory_percent‚87.64588038126627 disk.nvme0n1.in‚ 108.1171875 network.sent‚ 1.8811138e+07 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 108.1171875 disk.drbd2078.in‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51385934776103% disk./.usageGB‚1260.8594207763672 disk.drbd2111.in‚0 disk.rbd1.out‚ 98.9072265625¿­ÞOß-‚(:“ ¬‘‹Óچγ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.mem:u?OoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚115.658' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.55266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YªÅÃNà-‚(:à  »‘‹Ó•Ó³! disk.dm-0.out‚51095.248046875 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 108.1171875$ disk.nvme0n1.out‚51095.248046875 network.sent‚ 1.8871183e+07 disk.sda1.out‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 network.recv‚ 1.40576e+06 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1550.in‚0 cpu‚25.337421492471336) disk./.usagePercent‚70.51385934776103 disk.drbd2111.out‚0 disk.drbd1285.in‚0) proc.memory.percent‚11.92959149678548 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0) proc.memory.availableMB‚ 2725.59765625 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 818.29296875 disk.sda2.out‚ 818.29296875 disk.drbd1938.in‚0 disk.dm-0.in‚ 108.1171875 disk.rbd1.in‚0% disk./.usageGB‚1260.8594207763672 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.rbd1.out‚ 99.0634765625 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd2078.out‚0# memory_percent‚88.9095147450765—ÝHÉžá-‚(:“ »‘‹Ó±œ²³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚283.642' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.54733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ­²žâ-‚(:“ Ê‘‹Óšñ׳ gpu.0.gpu‚37.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.525' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.84166666666665 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EsJã-‚(:¿  Ê‘‹Ó¥÷³% disk./.usageGB‚1260.8594207763672 disk.sda2.out‚ 818.546875 disk.drbd1931.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625# proc.memory.rssMB‚ 2931.81640625 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 cpu‚25.337379385144366 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0$ memory_percent‚90.06840387980142 disk.drbd2118.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚51095.388671875 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 99.0634765625 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 818.546875 disk.drbd1673.in‚0 disk.drbd1584.in‚0( proc.memory.availableMB‚ 2440.7890625 network.sent‚ 1.887418e+07 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0! disk.dm-0.out‚51095.388671875 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51385934776103 disk.drbd1289.in‚0 disk.dm-0.in‚ 108.1171875) proc.memory.percent‚11.92959149678548 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 108.1171875 network.recv‚ 1.406673e+06§Ó)ÂMä-‚(:  Ù‘‹Óꎇ´ disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 819.015625 disk.dm-0.in‚ 108.1171875 disk.drbd1965.out‚0 network.recv‚ 1.407629e+06 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 99.0634765625 disk.drbd1356.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2931.81640625 cpu‚25.33731231153014 disk.drbd1285.in‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8595352172852 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0$ memory_percent‚91.40698115030924 disk.drbd1356.in‚0) proc.memory.percent‚11.92959149678548 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.sda1.out‚0 network.sent‚ 1.887718e+07 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51386574789625 disk.drbd1552.in‚0 disk.sda.out‚ 819.015625 disk.drbd2041.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 108.1171875 disk.sda.in‚ 0.03515625 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚51095.71337890625 disk.drbd1965.in‚0& disk.nvme0n1.out‚51095.71337890625( proc.memory.availableMB‚ 2111.8203125 disk.drbd1550.out‚0wR“å-‚(:ˆ Ù‘‹Ó¶Ÿç³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.903' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.301 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NnvMæ-‚(:  è‘‹Óûöϳ disk.sda.out‚ 819.28125 disk.drbd1673.in‚0 disk.sda2.out‚ 819.28125 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1941.91015625 disk.drbd2118.out‚0 disk.dm-0.in‚ 108.1171875 disk.drbd2078.in‚0 disk.drbd1285.out‚0 network.sent‚ 1.8880216e+07 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd2118.in‚0# disk.dm-0.out‚51095.99462890625 disk.drbd2111.out‚0& disk.nvme0n1.out‚51095.99462890625 disk.drbd1226.in‚0$ memory_percent‚92.09834734598795# proc.memory.rssMB‚ 2931.81640625 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚ 99.095703125 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.51386361451785% disk./.usageGB‚1260.8594970703125 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0) proc.memory.percent‚11.92959149678548 cpu‚25.337259559792813 disk.sda1.in‚0 disk.drbd2078.out‚0 network.recv‚ 1.408728e+06 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 108.1171875æøôžç-‚(:“ è‘‹Óà†¨³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚274.115' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.37166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖCk3Pè-‚(:Å  ÷‘‹Óþ‡ß³ proc.cpu.threads‚17# proc.memory.rssMB‚ 2931.81640625 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0) proc.memory.percent‚11.92959149678548 disk.drbd2041.in‚0 disk.drbd1965.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0# disk.dm-0.out‚51096.84912109375 cpu‚25.33721811081917% disk./.usageGB‚1260.8598442077637 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.51388302826136 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 99.189453125 network.recv‚ 1.40956e+06 disk.sda2.out‚ 819.58203125 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 108.1171875( proc.memory.availableMB‚ 1941.9140625 network.sent‚ 1.8883204e+07 disk.drbd1817.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚51096.84912109375 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0$ memory_percent‚92.09833145141602 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 819.58203125 disk.drbd1550.in‚0;w;“é-‚(:ˆ ÷‘‹Ó¿‹À³ gpu.0.gpu‚53.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.616' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.872 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h„[4ê-/ _wandbruntime‚19638  _runtime‚19638ÓÍÌ Éë-à % _timestamp‚1.7848588748832304e+09 _runtime‚19638.240461848 _step‚898 & lr-AdamW/pg1‚1.503960692977224e-05 & lr-AdamW/pg2‚1.503960692977224e-05 trainer/global_step‚10799Éa6´Îì-È _step‚898 & lr-AdamW/pg1‚1.503960692977224e-05 & lr-AdamW/pg2‚1.503960692977224e-05 trainer/global_step‚10799 % _timestamp‚1.7848588748832304e+09 _runtime‚19638.240461848‚RT4@4í-/ _wandbruntime‚19638  _runtime‚19638\ó»°(î-P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3109614137034752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 % _timestamp‚1.7848588748843398e+09 _runtime‚19638.241812263 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3109614110861312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11269888549888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13784091235840 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93766430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28778604 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28778599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3120885371020800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3120883999411200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13784144600064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61688661 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93766424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11271233986048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194056555 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100290125 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚850786296996864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚369 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32909688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚864570441596928 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3119640356415018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100290125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61688287 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11252362739770 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3109614137034752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚864570020141568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚850785928905728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3120883999411200 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3108387968247280 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚05ú°(ï-”P RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11252362739770 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3109614137034752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚864570020141568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚850785928905728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3120883999411200 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3108387968247280 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3109614137034752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 % _timestamp‚1.7848588748843398e+09 _runtime‚19638.241812263 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3109614110861312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11269888549888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13784091235840 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93766430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28778604 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28778599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3120885371020800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3120883999411200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13784144600064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61688661 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93766424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11271233986048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194056555 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100290125 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚850786296996864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚369 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32909688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚864570441596928 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3119640356415018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100290125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61688287 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ƒ!Tx4ð-/ _wandbruntime‚19639  _runtime‚19639—ãhHñ-™N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3109893046325248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3121165302826496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11270911057920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194074018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93774797 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100299221 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_ÉkŽ×"train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194074018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850864515271680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚900 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194063934 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11272256501248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61694280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32913036 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11272256501248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28781244 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93774791 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3119920178990546 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3121165302826496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61693910 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11270911057920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3109893046325248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194063934 trainer/global_step‚10799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3119918808951180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚19639.715587084 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194074018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850864515271680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚900 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194063934 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11272256501248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61694280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32913036 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11272256501248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28781244 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93774791 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3119920178990546 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3121165302826496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61693910 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11270911057920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3109893046325248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194063934 trainer/global_step‚10799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3119918808951180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 _runtime‚19639.715587084„ÕFJ‹4ó-/ _wandbruntime‚19639  _runtime‚19639šH“? ô-„ 2loss/SRqqt2/assignment_loss‚0.5078836537664787 1loss/SRqqt1/detection_loss‚0.8902904838208412 1loss/SRqqt2/detection_loss‚0.5980620746734603 /loss/FBt1/detection_loss‚0.4856844463837192 0loss/FRt2/assignment_loss‚0.6725145884922572 /loss/FRt2/detection_loss‚1.4786948229319283 /loss/FBt2/detection_loss‚0.4942733082742222 epoch‚38 $loss/total_loss‚7.72242824055236 .loss/FRt1/assignment_loss‚1.65846846673026 2loss/SRqqt1/assignment_loss‚0.4204573091928544 2loss/FBt2/assignment_loss‚0.006887945373745625 /loss/FRt1/detection_loss‚0.5072185802124295 % _timestamp‚1.7848588763967643e+09 _step‚901 3loss/FBt1/assignment_loss‚0.0019925607001626436 trainer/global_step‚10799 _runtime‚19639.754004163¬T;gõ-‰ $loss/total_loss‚7.72242824055236 .loss/FRt1/assignment_loss‚1.65846846673026 2loss/SRqqt1/assignment_loss‚0.4204573091928544 2loss/FBt2/assignment_loss‚0.006887945373745625 /loss/FRt1/detection_loss‚0.5072185802124295 % _timestamp‚1.7848588763967643e+09 _step‚901 3loss/FBt1/assignment_loss‚0.0019925607001626436 trainer/global_step‚10799 _runtime‚19639.754004163 2loss/SRqqt2/assignment_loss‚0.5078836537664787 1loss/SRqqt1/detection_loss‚0.8902904838208412 1loss/SRqqt2/detection_loss‚0.5980620746734603 /loss/FBt1/detection_loss‚0.4856844463837192 0loss/FRt2/assignment_loss‚0.6725145884922572 /loss/FRt2/detection_loss‚1.4786948229319283 /loss/FBt2/detection_loss‚0.4942733082742222 epoch‚38…#Ž(žö-‚(:“ †’‹Óò€²³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚280.052' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.35066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®_¥P÷-‚(:Å  †’‹ÓÝÐг) disk./.usagePercent‚70.51388302826136 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚51097.01708984375 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0% disk./.usageGB‚1260.8598442077637 disk.drbd2111.out‚0 proc.cpu.threads‚17) proc.memory.percent‚11.92959149678548$ memory_percent‚92.09850629170737# proc.memory.rssMB‚ 2931.81640625 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 99.251953125 disk.drbd1510.in‚0 disk.drbd1550.in‚0) proc.memory.availableMB‚ 1941.87109375 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1817.out‚0 disk.sda2.out‚ 819.8828125 disk.drbd1931.in‚0 disk.drbd1938.out‚0 cpu‚25.33716465580801 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 108.1171875 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 network.recv‚ 1.410501e+06 disk.drbd1552.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.sda.out‚ 819.8828125 network.sent‚ 1.8886202e+07# disk.dm-0.out‚51097.01708984375 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.sda1.out‚0¨ R“ø-‚(:ˆ •’‹Ó͇ä³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.664' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.888 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0戎ÊQù-‚(:Æ  •’‹Óîðÿ³" disk.dm-0.out‚51097.4736328125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 108.1171875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 cpu‚25.337122663685175 network.sent‚ 1.8946739e+07 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd1584.out‚0 disk.drbd2041.out‚0% disk.nvme0n1.out‚51097.4736328125 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0) proc.memory.percent‚11.92959149678548 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1510.out‚0 network.recv‚ 1.412231e+06 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 820.30859375 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1941.86328125) disk./.usagePercent‚70.51388516163976 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1226.in‚0$ memory_percent‚92.09853808085123 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 99.251953125% disk./.usageGB‚1260.8598823547363 disk.drbd1552.out‚0 disk.sda.out‚ 820.30859375 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.sda2.in‚ 0.03515625_-½Ÿú-‚(:” ¤’‹Óš¾Á³ gpu.0.gpu‚30.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚109.504' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.501333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÌ…™Kû-‚(:À  ¤’‹Óä¢à³ disk.drbd2078.in‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1550.out‚0' proc.memory.availableMB‚ 1942.109375 network.sent‚ 1.8949245e+07 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0) proc.memory.percent‚11.92959149678548 cpu‚25.33705704263494# proc.memory.rssMB‚ 2931.81640625 disk.drbd1828.out‚0" disk.dm-0.out‚51097.6962890625 network.recv‚ 1.413151e+06 disk.drbd1938.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17# disk./.usageGB‚1260.85986328125 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.sda.out‚ 820.58984375 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 820.58984375 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51388409495057 disk.drbd1285.in‚0% disk.nvme0n1.out‚51097.6962890625 disk.dm-0.in‚ 108.1171875 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0" memory_percent‚92.097536722819 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 99.2841796875 disk.drbd1226.out‚0 disk.drbd2118.out‚0£Ä7žü-‚(:“ ³’‹ÓÚ†Á³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚270.702' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.23400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¬þ=Ný-‚(:à  ³’‹Ó®Õá³ disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0" disk.dm-0.out‚51098.8779296875 disk.drbd1356.in‚0 disk.drbd1817.out‚0# disk./.usageGB‚1260.85986328125 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd2111.in‚0 network.recv‚ 1.414079e+06 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 cpu‚25.337003493644612 disk.sda.out‚ 820.89453125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 820.89453125 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0) proc.memory.percent‚11.92959149678548 network.sent‚ 1.8952338e+07 disk.drbd1552.in‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51388409495057% disk.nvme0n1.out‚51098.8779296875$ memory_percent‚92.09855397542319 disk.drbd2041.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 99.3779296875 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1941.859375 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.dm-0.in‚ 108.1171875(Q&”þ-‚(:‰ Â’‹ÓΫâ³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.783' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.261 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tN!«Nÿ-‚(:à  Â’‹Ó¬´ÿ³ disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 network.recv‚ 1.415118e+06 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 821.24609375 disk.drbd1965.out‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0% disk.nvme0n1.out‚51099.1005859375 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 108.1171875" disk.dm-0.out‚51099.1005859375 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2931.81640625 network.sent‚ 1.8955335e+07 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1817.in‚0) proc.memory.percent‚11.92959149678548# disk./.usageGB‚1260.85986328125 disk.drbd1965.in‚0 disk.drbd1584.in‚0' proc.memory.availableMB‚ 1941.859375) disk./.usagePercent‚70.51388409495057 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 821.24609375$ memory_percent‚92.09855397542319 cpu‚25.336962089202203 disk.drbd1828.out‚0 disk.rbd1.out‚ 99.4404296875 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 108.1171875»{¸˜Ÿ€.‚(:” Ñ’‹ÓÏ÷׳ gpu.0.gpu‚96.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.545' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.181666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#,7;.‚(:  Ñ’‹Ó¹ˆø³ network.sent‚ 1.8958534e+07 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2118.in‚0$ disk./.usageGB‚1260.859848022461 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0$ memory_percent‚92.09855397542319# proc.memory.rssMB‚ 2931.81640625 disk.drbd1938.out‚0 disk.rbd1.out‚ 99.4404296875 disk.drbd1226.in‚0 disk.drbd1817.out‚0 network.recv‚ 1.416156e+06& disk.nvme0n1.out‚51100.64404296875 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.sda.out‚ 821.828125) proc.memory.percent‚11.92959149678548 proc.cpu.threads‚17) disk./.usagePercent‚70.51388324159919 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 cpu‚25.336896229505165 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1510.in‚0 disk.dm-0.in‚ 108.1171875' proc.memory.availableMB‚ 1941.859375 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.sda2.out‚ 821.828125# disk.dm-0.out‚51100.64404296875 disk.drbd1965.out‚0 disk.drbd211º¯áR51.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0q¥Õ4‚./ _wandbruntime‚19725  _runtime‚19725_­hl˃.Å ' lr-AdamW/pg1‚1.4593853692325524e-05 ' lr-AdamW/pg2‚1.4593853692325524e-05 trainer/global_step‚10849 % _timestamp‚1.7848589618257337e+09 _runtime‚19725.183821193 _step‚902³ÏsÜЄ.Ê trainer/global_step‚10849 % _timestamp‚1.7848589618257337e+09 _runtime‚19725.183821193 _step‚902 ' lr-AdamW/pg1‚1.4593853692325524e-05 ' lr-AdamW/pg2‚1.4593853692325524e-05†KØŽÎ4…./ _wandbruntime‚19725  _runtime‚19725?çíi(†.P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28910599 trainer/global_step‚10849 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚378 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100746077 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11322423542272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194920787 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61969579 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11302122011292 _step‚903 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33059358 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3134880653492224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3123559575386112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3122328082364048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11321078106112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _runtime‚19725.184061476 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94184780 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11321078106112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _runtime‚19725.184061476 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94184780 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100746077 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11322423542272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194920787 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61969579 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11302122011292 _step‚903 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33059358 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3134880653492224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3123559575386112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653‡’¹üŸ4ˆ./ _wandbruntime‚19726  _runtime‚19726/«À'‰.™N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194948320 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3122606884137824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3133911371562154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33062310 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚868641908197888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3123838484676608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194938236 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3122606858709856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚10849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94193141 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11323446057472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3123838510850048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94193141 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61975181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94193147 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11322100614144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11322100614144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11303142812932 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3123838484676608 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚854794201660416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13847659471872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100755173 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94193147 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100745095 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194948320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3135160585290752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3135161956907520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28913239 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005hï1Š.žN XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3123838484676608 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚854794201660416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13847659471872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100755173 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94193147 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100745095 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194948320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3135160585290752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3135161956907520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28913239 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194948320 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3122606884137824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3133911371562154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33062310 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚868641908197888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3123838484676608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194938236 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3122606858709856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚10849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94193141 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11323446057472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3123838510850048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94193141 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61975181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94193147 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11322100614144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11322100614144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11303142812932 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ˆf«Î4‹./ _wandbruntime‚19726  _runtime‚19726«–ŠØ Œ.… 2loss/FBt1/assignment_loss‚5.45072725618025e-05 1loss/SRqqt1/detection_loss‚0.8477255579728792 /loss/FRt1/assignment_loss‚1.585133740883375 1loss/SRqqt1/assignment_loss‚0.474814513956596 3loss/SRqqt2/assignment_loss‚0.48114446210048006 /loss/FRt2/detection_loss‚1.3757704547482508 1loss/SRqqt2/detection_loss‚0.5466075016606503 /loss/FBt1/detection_loss‚0.7304613061423879 %loss/total_loss‚7.635064434615155 trainer/global_step‚10849 /loss/FRt2/assignment_loss‚0.662005537893714 2loss/FBt2/assignment_loss‚0.057517428844876047 /loss/FRt1/detection_loss‚0.5017904020675107 epoch‚38 % _timestamp‚1.7848589633406875e+09 _step‚905 0loss/FBt2/detection_loss‚0.37203902107187364 _runtime‚19726.697877067~ðÃ&.Š 2loss/FBt1/assignment_loss‚5.45072725618025e-05 1loss/SRqqt1/detection_loss‚0.8477255579728792 /loss/FRt1/assignment_loss‚1.585133740883375 1loss/SRqqt1/assignment_loss‚0.474814513956596 3loss/SRqqt2/assignment_loss‚0.48114446210048006 /loss/FRt2/detection_loss‚1.3757704547482508 1loss/SRqqt2/detection_loss‚0.5466075016606503 /loss/FBt1/detection_loss‚0.7304613061423879 %loss/total_loss‚7.635064434615155 trainer/global_step‚10849 /loss/FRt2/assignment_loss‚0.662005537893714 2loss/FBt2/assignment_loss‚0.057517428844876047 /loss/FRt1/detection_loss‚0.5017904020675107 epoch‚38 % _timestamp‚1.7848589633406875e+09 _step‚905 0loss/FBt2/detection_loss‚0.37203902107187364 _runtime‚19726.697877067‰í×Db‘Ž.‚(:† à’‹ÓûÀѳ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.88' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.96 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦x©†J.‚(:¿  à’‹Ó“§ñ³ disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.dm-0.in‚ 108.1171875 network.sent‚ 1.8961567e+07 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 network.recv‚ 1.41707e+06 disk.drbd1938.out‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 99.47265625 disk.drbd1510.in‚0) disk./.usagePercent‚70.51388324159919 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1226.in‚0 disk.drbd1552.in‚0$ disk./.usageGB‚1260.859848022461 disk.drbd1552.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1941.79296875 disk.sda2.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.dm-0.out‚51101.04296875 disk.drbd1931.out‚0 cpu‚25.336855737446765 disk.drbd1550.in‚0$ memory_percent‚92.09882418314615 disk.sda1.out‚0 disk.sda.out‚ 822.05859375 disk.drbd2041.out‚0# disk.nvme0n1.out‚51101.04296875 disk.drbd1550.out‚0) proc.memory.percent‚11.92959149678548 disk.drbd1817.in‚0 disk.sda2.out‚ 822.05859375 disk.drbd2041.in‚0o²OГ.‚(:ˆ ï’‹Óҭϳ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.505' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.835 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òwþ@N‘.‚(:à  ï’‹ÓÒöì³ disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1584.out‚0 network.sent‚ 1.9021645e+07 disk.drbd1550.out‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2111.out‚0& disk.nvme0n1.out‚51101.64697265625 disk.sda.out‚ 822.3359375 disk.sda2.in‚ 0.03515625) proc.memory.availableMB‚ 1941.79296875 disk.drbd1226.in‚0 disk.drbd2041.in‚0 disk.dm-0.in‚ 108.1171875# disk.dm-0.out‚51101.64697265625 disk.drbd2078.out‚0 disk.rbd1.out‚ 99.56640625 cpu‚25.336802626446747) disk./.usagePercent‚70.51388324159919 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1938.in‚0 network.recv‚ 1.41868e+06 disk.drbd1965.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.09882418314615) proc.memory.percent‚11.92959149678548 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0$ disk./.usageGB‚1260.859848022461 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 822.3359375 disk.drbd1584.in‚0 disk.drbd1938.out‚0¡²Òh’.‚(:’ þ’‹Ó÷¾Ò³ gpu.0.gpu‚69.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚114.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.17333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dºDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3148877307573248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚872553715339776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚858644933592064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195795089 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101191965 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3147621371518980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62250967 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚10899 % _timestamp‚1.7848590484871807e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3148878679182848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94603124 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101191965 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3148877307573248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3137505039910912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11372267662336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 _step‚907 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13908730480640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29042599 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94603124 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600Þç, p.”P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3137505039910912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195805159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3137505066084352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3136268171052848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29042604 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3148878679182848 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3148877307573248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚872553715339776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚858644933592064 =7DeviceStatsMonitor.on_train_batch_ÙÈR1ª!start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195795089 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101191965 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3147621371518980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62250967 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 trainer/global_step‚10899 % _timestamp‚1.7848590484871807e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3148878679182848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94603124 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101191965 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3148877307573248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3137505039910912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11372267662336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 _step‚907 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13908730480640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29042599 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94603124 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚872633129384960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195822622 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11354246695740 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3137783949201408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62256601 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29045239 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3149158610988544 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101201047 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3149158610988544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3137783949201408 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11374635613696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.784859049958767e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 trainer/global_step‚10899 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94611491 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101211125 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚872633552930304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3136546947398656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11355591307138 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚373 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62256223 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33211357 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33210984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195812538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚378 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101201047 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94611497 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11374635613696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195812538 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94611491 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚554542089Ñwm#' .N BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62256223 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33211357 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33210984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195812538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚378 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101201047 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94611497 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11374635613696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195812538 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94611491 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3137783975374848 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101211125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29045244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚858723151866880 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13910032972288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚872633129384960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195822622 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11354246695740 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3137783949201408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62256601 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29045239 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3149158610988544 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101201047 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3149158610988544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3137783949201408 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11374635613696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.784859049958767e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 trainer/global_step‚10899 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94611491 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101211125 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚872633552930304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3136546947398656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11355591307138 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚373Œ¶Û( 4¡./ _wandbruntime‚19813  _runtime‚19813 eÁµ ¢.† /loss/FRt2/detection_loss‚1.4501010654127726 0loss/SRqqt1/detection_loss‚0.816067620172029 1loss/SRqqt2/detection_loss‚0.6028348812205246 /loss/FBt1/detection_loss‚0.7558933494146913 trainer/global_step‚10899 0loss/FRt2/assignment_loss‚0.6928155384687188 2loss/SRqqt1/assignment_loss‚0.5001229291859245 2loss/SRqqt2/assignment_loss‚0.5270074161961508 3loss/FBt1/assignment_loss‚2.375644728555163e-06 1loss/FBt2/assignment_loss‚0.05263491925556891 epoch‚38 _runtime‚19813.355051084 _step‚909 /loss/FRt1/detection_loss‚0.5477968079054261 0loss/FBt2/detection_loss‚0.38041497069341995 &loss/total_loss‚7.8625616008526125 % _timestamp‚1.7848590499978468e+09 /loss/FRt1/assignment_loss‚1.536869727282657Àêü£.‹ 1loss/FBt2/assignment_loss‚0.05263491925556891 epoch‚38 _runtime‚19813.355051084 _step‚909 /loss/FRt1/detection_loss‚0.5477968079054261 0loss/FBt2/detection_loss‚0.38041497069341995 &loss/total_loss‚7.8625616008526125 % _timestamp‚1.7848590499978468e+09 /loss/FRt1/assignment_loss‚1.536869727282657 /loss/FRt2/detection_loss‚1.4501010654127726 0loss/SRqqt1/detection_loss‚0.816067620172029 1loss/SRqqt2/detection_loss‚0.6028348812205246 /loss/FBt1/detection_loss‚0.7558933494146913 trainer/global_step‚10899 0loss/FRt2/assignment_loss‚0.6928155384687188 2loss/SRqqt1/assignment_loss‚0.5001229291859245 2loss/SRqqt2/assignment_loss‚0.5270074161961508 3loss/FBt1/assignment_loss‚2.375644728555163e-06æ‰0#¤.‚(:’ «“‹Ó£ì˳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.93' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.97666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aps@M¥.‚(:  «“‹Ó¡‚ì³) disk./.usagePercent‚70.51391076218067 disk.dm-0.in‚ 108.1171875 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1673.out‚0$ memory_percent‚92.09993680318198% disk./.usageGB‚1260.8603401184082 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1938.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0! disk.dm-0.out‚51103.458984375 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd2078.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 823.6640625 disk.drbd1285.in‚0 network.recv‚ 1.422557e+06 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1941.51953125 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 108.1171875 cpu‚25.336603219628405 disk.sda2.in‚ 0.03515625 disk.rbd1.out‚ 100.05078125 network.sent‚ 1.9033667e+07 disk.drbd1356.in‚0 disk.drbd2111.out‚0$ disk.nvme0n1.out‚51103.458984375 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1584.in‚0 disk.drbd1510.in‚0 disk.sda.out‚ 823.6640625) proc.memory.percent‚11.92959149678548.E5¦`¦.‚(:Á  º“‹Ó–ñ³ disk.sda.out‚ 823.98828125 proc.cpu.threads‚17$ memory_percent‚92.09969838460287' proc.memory.availableMB‚ 1941.578125 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 100.11328125 cpu‚25.33658806201518 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.d“éîìrbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8600578308105 disk.drbd1931.in‚0 disk.rbd1.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1510.in‚0! disk.dm-0.out‚51103.619140625 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1828.in‚0 network.recv‚ 1.424593e+06) disk./.usagePercent‚70.51389497518045 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 823.98828125 disk.drbd1289.out‚0 disk.drbd2078.in‚0 network.sent‚ 1.9094286e+07 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2931.81640625 disk.sda1.in‚0 disk.dm-0.in‚ 108.1171875 disk.drbd1673.out‚0) proc.memory.percent‚11.92959149678548 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚51103.619140625 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0{õä6“§.‚(:ˆ º“‹ÓÕàÒ³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚209.826' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚69.942 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—é§è¨.‚(:’ É“‹ÓÏŽÀ³ gpu.0.gpu‚13.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚60.0 gpu.0.powerWatts‚108.283' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.09433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º6,mO©.‚(:Ä  É“‹Óñ¨ß³ disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.09980964660645 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0# disk.dm-0.out‚51104.65673828125 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1828.in‚0 network.sent‚ 1.9096747e+07) disk./.usagePercent‚70.51389497518045 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚51104.65673828125 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 108.1171875 disk.rbd1.in‚0 network.recv‚ 1.425561e+06 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0# proc.memory.rssMB‚ 2931.81640625 disk.drbd1817.in‚0 disk.drbd1673.out‚0) proc.memory.percent‚11.92959149678548% disk./.usageGB‚1260.8600578308105 disk.sda.out‚ 824.296875 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 cpu‚25.336523096909463 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 100.11328125 disk.nvme0n1.in‚ 108.1171875 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.sda2.out‚ 824.296875 disk.drbd1226.out‚0) proc.memory.availableMB‚ 1941.550781256ä§Ö”ª.‚(:‰ Ø“‹Óª©É³ gpu.0.gpu‚95.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚308.217' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.739 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð7EQ«.‚(:Æ  Ø“‹Ó¸é³# disk.dm-0.out‚51104.84423828125 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.dm-0.in‚ 108.1171875 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 824.6796875 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0& disk.nvme0n1.out‚51104.84423828125 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1550.out‚0% disk./.usageGB‚1260.8600769042969 disk.drbd1938.out‚0 disk.drbd1510.in‚0 cpu‚25.337852564946413$ memory_percent‚92.10592905680338 network.recv‚ 1.426648e+06 disk.rbd1.out‚100.1455078125 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625' proc.memory.availableMB‚ 1940.046875 network.sent‚ 1.9099879e+07 disk.drbd1828.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 824.6796875 disk.sda2.in‚ 0.03515625 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.934947967529297) disk./.usagePercent‚70.51389604186966 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0" proc.memory.rssMB‚ 2933.1328125A‡4¬./ _wandbruntime‚19867  _runtime‚19867òBt;­.ûu ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6317436099052429 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8052195906639099 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3645291328430176 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 (particle/accuracy‚0.8559163808822632 trainer/global_step‚10919 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.37435799837112427 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6515722870826721 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8835787773132324 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6982516050338745 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5516145825386047 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8603895902633667 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7049517035484314 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4011378884315491 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7280973196029663 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26621174812316895 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7944141626358032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29360872507095337 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7232781052589417 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17114026844501495 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36355486512184143 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9910191297531128 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978538751602173 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8565707802772522 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28192049264907837 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8135952353477478 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02574361488223076 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6576099991798401 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4600771963596344 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.1148163229227066 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6788836121559143 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7787429094314575 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 epoch‚38 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6642147898674011 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_3‚0.07890298217535019 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051903191953897476 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3463423550128937 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13972266018390656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7430186867713928 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7341591715812683 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35110124945640564 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9224410653114319 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.718036949634552 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7183760404586792 *jet/accuracy_1_of_3‚0.9392528533935547 particle/accuracy_2_of_3‚1 'particle/f_score‚0.6556368470191956 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7516502737998962 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1538788378238678 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032484717667102814 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35142797231674194 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2681901156902313 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9970049262046814 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2241346836090088 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8465908765792847 +particle/sensitivity‚0.6393434405326843 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6445707082748413 *jet/accuracy_1_of_1‚0.3733825385570526 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967465400695801 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6674624085426331 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2900643050670624 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.701506495475769 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 +jet/accuracy_2_of_2‚0.17207686603069305 particle/accuracy_1_of_4‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7130712866783142 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39235901832580566 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.993764340877533 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2946244180202484 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08578116446733475 +particle/specificity‚0.9150736927986145 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8543729782104492 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9957947134971619 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6230769753456116 _step‚910 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3426322638988495 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6351537704467773 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7056421041488647 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04232895374298096 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27885302901268005 /particle/accuracy_4_of_4‚0.9716639518737793 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.686955451965332 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7293074727058411 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.599910318851471 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47446465492248535 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16670678555965424 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03327728435397148 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 *jet/accuracy_1_of_4‚0.9794445633888245 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6102551817893982 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7390722632408142 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38984018564224243 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22763773798942566 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 _runtime‚19867.350424273 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6492215394973755 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.754175066947937 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8240440487861633 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8002412915229797 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06942812353372574 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6576099991798401 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30259519815444946 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3627961575984955 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6534838676452637 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5258560180664062 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.871780514717102 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2762686312198639 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8835787773132324 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3877193331718445 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9421166777610779 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8752089738845825 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8138304352760315 *jet/accuracy_2_of_3‚0.6065620183944702 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_4‚0.8435002565383911 7validation_average_jet_accuracy‚0.40750598907470703 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8620129227638245 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17075514793395996 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18385860323905945 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35142797231674194 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47243911027908325 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 )loss/val_total_loss‚7.810185691532044 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5827071070671082 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6768816709518433 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5155945420265198 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6688750386238098 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3052443563938141 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 % _timestamp‚1.7848591039923797e+09 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699678540229797 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7738984227180481 ,jet/accuracy_4_of_4‚0.018756305798888206 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2617018222808838 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35508885979652405 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025166111066937447 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 £ <.®.€v =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6317436099052429 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8052195906639099 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3645291328430176 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 (particle/accuracy‚0.8559163808822632 trainer/global_step‚10919 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.37435799837112427 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6515722870826721 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8835787773132324 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6982516050338745 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5516145825386047 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8603895902633667 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7049517035484314 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4011378884315491 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7280973196029663 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26621174812316895 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7944141626358032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29360872507095337 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7232781052589417 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17114026844501495 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36355486512184143 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9910191297531128 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9978538751602173 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8565707802772522 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28192049264907837 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8135952353477478 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02574361488223076 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6576099991798401 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9967843294143677 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4600771963596344 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.1148163229227066 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6788836121559143 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7787429094314575 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 epoch‚38 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6642147898674011 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_3‚0.07890298217535019 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051903191953897476 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3463423550128937 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13972266018390656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7430186867713928 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7341591715812683 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35110124945640564 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9224410653114319 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.718036949634552 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7183760404586792 *jet/accuracy_1_of_3‚0.9392528533935547 particle/accuracy_2_of_3‚1 'particle/f_score‚0.6556368470191956 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7516502737998962 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1538788378238678 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032484717667102814 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35142797231674194 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2681901156902313 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9970049262046814 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2241346836090088 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8980615735054016 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8465908765792847 +particle/sensitivity‚0.6393434405326843 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6445707082748413 *jet/accuracy_1_of_1‚0.3733825385570526 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967465400695801 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6674624085426331 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014519141055643559 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2900643050670624 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.701506495475769 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 +jet/accuracy_2_of_2‚0.17207686603069305 particle/accuracy_1_of_4‚1 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7130712866783142 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39235901832580566 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.993764340877533 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2946244180202484 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08578116446733475 +particle/specificity‚0.9150736927986145 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8543729782104492 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9982718229293823 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9957947134971619 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6230769753456116 _step‚910 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3426322638988495 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6351537704467773 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7056421041488647 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04232895374298096 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27885302901268005 /particle/accuracy_4_of_4‚0.9716639518737793 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.686955451965332 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7293074727058411 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.599910318851471 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47446465492248535 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16670678555965424 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03327728435397148 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975920915603638 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 *jet/accuracy_1_of_4‚0.9794445633888245 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6102551817893982 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7390722632408142 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38984018564224243 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22763773798942566 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 _runtime‚19867.350424273 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6492215394973755 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.754175066947937 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8240440487861633 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8002412915229797 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06942812353372574 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014519141055643559 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6576099991798401 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30259519815444946 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3627961575984955 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.0194963719695Õ™‰Ê 8065 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6534838676452637 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5258560180664062 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.871780514717102 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2762686312198639 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8835787773132324 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3877193331718445 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9421166777610779 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8752089738845825 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9952381253242493 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8138304352760315 *jet/accuracy_2_of_3‚0.6065620183944702 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9904761910438538 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_4‚0.8435002565383911 7validation_average_jet_accuracy‚0.40750598907470703 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8620129227638245 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17075514793395996 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18385860323905945 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35142797231674194 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47243911027908325 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 )loss/val_total_loss‚7.810185691532044 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5827071070671082 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6768816709518433 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5155945420265198 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6688750386238098 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9971340894699097 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3052443563938141 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4469697177410126 % _timestamp‚1.7848591039923797e+09 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699678540229797 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7738984227180481 ,jet/accuracy_4_of_4‚0.018756305798888206 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2617018222808838 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35508885979652405 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025166111066937447Ž§Î‘Ÿ¯.‚(:” ç“‹Ó¨Û­³ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚62.0 gpu.0.powerWatts‚302.832' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.94399999999999 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vLaR°.‚(:Ç  ç“‹ÓŠ­Ï³ disk.dm-0.in‚ 108.1171875 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚100.3251953125$ memory_percent‚88.13918431599936 cpu‚25.33817619636537 disk.drbd1817.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2934.47265625 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0( proc.memory.availableMB‚ 2914.9140625) disk./.usagePercent‚70.51416228749503 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 network.sent‚ 1.9102882e+07 disk.drbd2078.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.940399805704754% disk./.usageGB‚1260.8648376464844 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.nvme0n1.in‚ 108.1171875 network.recv‚ 1.427618e+06 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.sda.out‚ 824.92578125 disk.sda2.out‚ 824.92578125 disk.drbd1931.out‚0" disk.dm-0.out‚52074.6904296875 disk.drbd2111.in‚0% disk.nvme0n1.out‚52074.6904296875 proc.cpu.threads‚17 disk.drbd1550.in‚0Œ™Å?R±.‚(:Ç  ö“‹Ó¶Ñé³" disk.dm-0.out‚52074.9130859375$ memory_percent‚88.13910484313965 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 cpu‚25.33813436372493 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 network.recv‚ 1.429498e+06 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 108.1171875 disk.nvme0n1.in‚ 108.1171875 disk.drbd1226.in‚0) disk./.usagePercent‚70.51416122080583 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.940399805704754 disk.sda1.out‚0 disk.drbd1938.out‚0$ disk./.usageGB‚1260.864818572998 disk.sda1.in‚0 disk.sda.out‚ 825.27734375 disk.drbd1673.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2934.47265625 network.sent‚ 1.9157364e+07 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.rbd1.out‚100.3876953125 disk.drbd1510.in‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 2914.93359375 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 825.27734375% disk.nvme0n1.out‚52074.9130859375 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0ΔbµŸ².‚(:” ö“‹ÓáÏ̳ gpu.0.gpu‚38.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚336.328' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚112.10933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Lu¼ž³.‚(:“ …”‹ÓÓàÛ³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.924' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.64133333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\™YR´.‚(:Ç  …”‹ÓÏÞþ³ disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚100.3876953125 disk.sda2.in‚ 0.03515625 disk.drbd2078.out‚0) disk./.usagePercent‚70.51395044301907 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0$ memory_percent‚88.14013799031576 cpu‚25.338067616330736 network.sent‚ 1.915983e+07 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 2914.6796875# proc.memory.rssMB‚ 2934.47265625* proc.memory.percent‚11.940399805704754 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 825.63671875 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625" disk.dm-0.out‚53044.8935546875 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0% disk.nvme0n1.out‚53044.8935546875 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd2078.in‚0 network.recv‚ 1.430404e+06 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 825.63671875 disk.drbd1931.out‚0 disk.drbd1938.in‚0% disk./.usageGB‚1260.8610496520996 disk.dm-0.in‚ 108.1171875[ëÂcžµ.‚(:“ ””‹ÓÃÖ¿³ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚288.779' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.25966666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o!ÀO¶.‚(:Ä  ””‹Ó˜¼Ü³ disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.dm-0.in‚ 108.1171875 disk.drbd1510.out‚0$ memory_percent‚84.51248804728189 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0( proc.memory.availableMB‚ 3806.2109375 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 826.05859375* proc.memory.percent‚11.940399805704754 network.recv‚ 1.431419e+06 disk.drbd1817.out‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.8610153198242 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2934.47265625 cpu‚25.3380397126293 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0" disk.dm-0.out‚53045.0927734375 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 108.1171875( disk./.usagePercent‚70.5139485229785 disk.drbd1965.in‚0 disk.rbd1.out‚ 100.419921875 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 826.05859375% disk.nvme0n1.out‚53045.0927734375 disk.drbd1550.out‚0 network.sent‚ 1.9162881e+07Œ4ð¯4·./ _wandbruntime‚19922  _runtime‚19922Àß-˸.Å ' lr-AdamW/pg1‚1.3719695527470384e-05 ' lr-AdamW/pg2‚1.3719695527470384e-05 trainer/global_step‚10949 % _timestamp‚1.7848591595564713e+09 _runtime‚19922.913735293 _step‚911W^iYй.Ê ' lr-AdamW/pg1‚1.3719695527470384e-05 ' lr-AdamW/pg2‚1.3719695527470384e-05 trainer/global_step‚10949 % _timestamp‚1.7848591595564713e+09 _runtime‚19922.913735293 _step‚911 †•4º./ _wandbruntime‚19922  _runtime‚19922Ò˜wÃ(».P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11429075657728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚876601932316672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3154011181160448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62565895 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13978680913920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848591595575483e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3165441628427776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95110912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29181496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101705293 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101695229 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11411283227508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33384735 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3152767235464848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196806135 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101695229 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95110906 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3164178544120324 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3154011207333888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11430421093888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3165440256818176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29181491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33384404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚862623251402752 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3152767260892816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11409938616222 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3165441628427776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚876601519249920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ÚóŽ(¼.’P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11430421093888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3165440256818176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29181491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33384404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚862623251402752 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3152767260892816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11409938616222 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3165441628427776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚876601519249920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11429075657728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚331 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚876601932316672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3154011181160448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62565895 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13978680913920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7848591595575483e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3165441628427776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95110912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29181496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101705293 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101695229 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11411283227508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33384735 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3152767235464848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196806135 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101695229 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95110906 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3164178544120324 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3154011207333888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 413066752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005Åöà4½./ _wandbruntime‚19924  _runtime‚19924YÀ?ž¾.™N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3165721560233472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 % _timestamp‚1.7848591610285637e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11431443609088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101704311 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62571555 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3153046037238624 trainer/global_step‚10949 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚325 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11412304029260 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196823584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3154290116624384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13979961543680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11430098165760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62571880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196833668 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11430098165760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95119279 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101714389 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95119279 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3165721560233472 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3165720188616704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196833668 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0vÑ$'¿.žN YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11410959417862 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95119273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚876681388349952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101704311 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3154290116624384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101714389 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11412304029260 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196823584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3154290116624384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13979961543680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11430098165760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 410962432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62571880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196833668 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11430098165760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95119279 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101714389 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95119279 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3165721560233472 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3165720188616704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196833668 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3165721560233472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 % _timestamp‚1.7848591610285637e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11431443609088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101704311 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62571555 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3153046037238624 trainer/global_step‚10949 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚325 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10999 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3179436910899200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102151181 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197690507 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102151181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3179436910899200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚20009.58800237 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3166707324153648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29313496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚880593452260864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33533180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14041250651648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3167956671858688 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3167956645685248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3178169736691932 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3179438282508800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95529256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11461042499030 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11481610650112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚866552201609216 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62846316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102161245 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11462387110316 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚114816106501125«z(Ò.“P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62846316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102161245 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11462387110316 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11481610650112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197680437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29313491 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3167956645685248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10999 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3179436910899200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102151181 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197690507 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102151181 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3179436910899200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _runtime‚20009.58800237 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3166707324153648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29313496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚880593452260864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33533180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14041250651648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3167956671858688 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3167956645685248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3178169736691932 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3179438282508800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95529256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11461042499030 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11481610650112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚866552201609216 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976”Ï)ÖlQÓ.‚(:Æ  Óö¬ó³ disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 108.1171875 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0' proc.memory.availableMB‚ 2028.484375 disk.drbd1931.in‚0 network.sent‚ 1.9238157e+07 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 828.14453125% disk./.usageGB‚1260.8610153198242 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.sda.in‚ 0.03515625" disk.dm-0.out‚53047.3974609375 network.recv‚ 1.438205e+06 disk.drbd1356.in‚0% disk.nvme0n1.out‚53047.3974609375# proc.memory.rssMB‚ 2934.47265625 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 cpu‚25.337743702097278( disk./.usagePercent‚70.5139485229785 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 828.14453125 disk.rbd1.out‚100.7646484375 disk.drbd1285.out‚0* proc.memory.percent‚11.940399805704754 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 108.1171875 disk.drbd1673.in‚0$ memory_percent‚91.74607594807944 proc.cpu.threads‚17ýÔKŸÔ.‚(:” Ó¡—г gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.472' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.15733333333333 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ´ð¤4Õ./ _wandbruntime‚20011  _runtime‚20011k‘‰'Ö.™N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11463407912068 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3178448189228094 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11481287721984 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3168235554975744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197697886 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11482633165312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197707970 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14042498404864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚866630787975168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848592477033694e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95537629 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3168235554975744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102170341 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102160263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3166986125927424 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102160263 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚880673286380032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚20011.060985228 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3168235581149184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33536109 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3178449559267460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62851896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚866631156066304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3179718214314496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3166986151355392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11482633165312é'ª/×.žN KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11481287721984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3179716842697728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonˆÜ`õ$itor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3168235554975744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197697886 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11482633165312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197707970 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14042498404864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚866630787975168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848592477033694e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95537629 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3168235554975744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102170341 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102160263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3166986125927424 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102160263 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚880673286380032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 _runtime‚20011.060985228 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3168235581149184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33536109 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3178449559267460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62851896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚866631156066304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3179718214314496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3166986151355392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11482633165312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11463407912068 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3178448189228094 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11481287721984 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11532800206336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198564809 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95947612 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14103872886272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3192160929263540 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102607133 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3193434936589824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3181902136383488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198554739 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95947606 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3193434936589824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚870481519906816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848593329591515e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884585760884224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29445491 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33680941 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33680587 _step‚920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3180647438270416 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3181902110210048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102617197 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3181902110210048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3193433564980224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11532800206336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198564809 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95947612 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14103872886272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3192160929263540 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102607133 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3193434936589824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3181902136383488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198554739 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95947606 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3193434936589824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚870481519906816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848593329591515e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884585760884224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29445491 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33680941 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33680587 _step‚920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3180647438270416 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3181902110210048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102617197 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3181902110210048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3193433564980224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11532477278208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚884665181369344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198572188 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚20097.793341482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95955973 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11533822721536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95955979 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11513167183478 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198572188 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3193713496778752 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11533822721536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3180926240044192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚11049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63131684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198582272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102626293 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3182181019500544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95955973 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚870560106272768 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11532477278208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚884665181369344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198572188 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚20097.793341482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95955973 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11533822721536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95955979 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11513167183478 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198572188 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3193713496778752 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11533822721536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3180926240044192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚11049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63131684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198582272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102626293 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3182181019500544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95955973 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚870560106272768 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11583989762560 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11582644326400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚874411206295552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33828154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11582644326400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29577491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3194587526959216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199429041 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103063085 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3195847574734848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103073149 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11564594875932 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96365956 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103063085 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63405645 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚11099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96365956 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29577496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3206150751795894 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11563250264646 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚874410838204416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3207430219061248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96365962 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63406015 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14166351043584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚924 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3207430219061248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3195847600908288 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3195847574734848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3195847600908288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚365 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199439111Ïj·(‚/”P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3195847600908288 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3195847574734848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3195847600908288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚365 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199439111 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _runtime‚20182.995942024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11583989762560 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11582644326400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚874411206295552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33828154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11582644326400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29577491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3194587526959216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199429041 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103063085 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3195847574734848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103073149 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11564594875932 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96365956 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103063085 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63405645 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 trainer/global_step‚11099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96365956 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29577496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3206150751795894 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11563250264646 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚874410838204416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3207430219061248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96365962 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63406015 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14166351043584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 _step‚924 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3207430219061248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248œI‚¡«4ƒ// _wandbruntime‚20184  _runtime‚20184‰ ·"„/™N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103072167 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29580136 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63411217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3207711522476544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11585012277760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3196126510198784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96374323 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3194866328732992 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3196126510198784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33831456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888657387067392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 438225408 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14167594405888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11564271066286 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3194866303305024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103082245 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103072167 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 % _timestamp‚1.7848594211143746e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11583666834432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11565615677684 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33831086 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3207711522476544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199446490 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96374329 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚370 trainer/global_step‚11099 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3207710150859776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63411592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96374323 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199456574 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864Ýß¾y$'…/žN QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚874489424570368 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3196126484025344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29580131 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3206430574371310 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199446490 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63411217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3207711522476544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11585012277760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3196126510198784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96374323 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3194866328732992 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3196126510198784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33831456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888657387067392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 438225408 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14167594405888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11564271066286 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3194866303305024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103082245 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103072167 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 % _timestamp‚1.7848594211143746e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11583666834432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11565615677684 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33831086 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3207711522476544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199446490 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96374329 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚370 trainer/global_step‚11099 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3207710150859776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63411592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96374323 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199456574 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103072167 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29580136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3208527615648016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11635179318784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚878340156502016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200303343 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3209793065433088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96784306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚11149 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚892569074116608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29709496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3209793065433088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96784312 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3209793039259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3221426873142272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _runtime‚20269.872177852 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103519037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33976898 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11614354147454 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63686394 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63686013 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3221428244751872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3221428244751872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103529101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103529101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11615698758740 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3209793039259648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3220141944367502 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286ú‚ùŒ(š/”P ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3220143314406756 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11635179318784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚878340156502016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200303343 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3209793065433088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96784306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚11149 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚892569074116608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29709496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3209793065433088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96784312 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3209793039259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3221426873142272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _runtime‚20269.872177852 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103519037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33976898 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11614354147454 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63686394 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63686013 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3221428244751872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3221428244751872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103529101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103529101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11615698758740 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3209793039259648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3220141944367502 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3208527615648016 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63691614 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚878419110959104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103538197 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848595079898026e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200330876 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3221706804940800 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚892649334679040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11636201833984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3221708176557568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11636201833984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3208806417421792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103528119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚878418742867968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200320792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96792679 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3210071948550144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11634856390656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96792673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14230223719936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚929 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11634856390656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200330876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200320792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14230168265728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63691614 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚878419110959104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103538197 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848595079898026e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200330876 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3221706804940800 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚892649334679040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11636201833984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3221708176557568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11636201833984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3208806417421792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103528119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚878418742867968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200320792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96792679 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3210071948550144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11634856390656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96792673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14230223719936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚929 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚367 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11634856390656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200330876¡sÁØß4ž// _wandbruntime‚20271  _runtime‚20271…{Ÿ/ˆ 3loss/SRqqt2/assignment_loss‚0.48308209059383933 .loss/FRt2/detection_loss‚1.471644633955344 1loss/SRqqt1/detection_loss‚0.8269266891105461 0loss/FBt1/detection_loss‚0.44155848361551764 0loss/FRt1/assignment_loss‚1.5571275703235021 0loss/FRt2/assignment_loss‚0.6281883260538411 3loss/SRqqt1/assignment_loss‚0.49765596561120157 .loss/FRt1/detection_loss‚0.507926142397585 epoch‚39 2loss/FBt1/assignment_loss‚0.001020875605245269 2loss/FBt2/assignment_loss‚0.019460502430513723 1loss/SRqqt2/detection_loss‚0.5648518839744162 &loss/total_loss‚7.4343885278178465 trainer/global_step‚11149 % _timestamp‚1.7848595080285552e+09 _runtime‚20271.385788626 /loss/FBt2/detection_loss‚0.4349453641462948 _step‚930± ) / 0loss/FRt1/assignment_loss‚1.5571275703235021 0loss/FRt2/assignment_loss‚0.6281883260538411 3loss/SRqqt1/assignment_loss‚0.49765596561120157 .loss/FRt1/detection_loss‚0.507926142397585 epoch‚39 2loss/FBt1/assignment_loss‚0.001020875605245269 2loss/FBt2/assignment_loss‚0.019460502430513723 1loss/SRqqt2/detection_loss‚0.5648518839744162 &loss/total_loss‚7.4343885278178465 trainer/global_step‚11149 % _timestamp‚1.7848595080285552e+09 _runtime‚20271.385788626 /loss/FBt2/detection_loss‚0.4349453641462948 _step‚930 3loss/SRqqt2/assignment_loss‚0.48308209059383933 .loss/FRt2/detection_loss‚1.471644633955344 1loss/SRqqt1/detection_loss‚0.8269266891105461 0loss/FBt1/detection_loss‚0.44155848361551764¢{¡tVQ¡/‚(:Æ  ü–‹ÓóÁá³ disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚101.8466796875# proc.memory.rssMB‚ 2934.47265625 disk.dm-0.in‚ 116.953125# disk.dm-0.out‚53054.18798828125 disk.drbd2111.in‚0 disk.drbd1285.out‚0 network.recv‚ 1.459269e+06 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0) disk./.usagePercent‚70.51409188600755 disk.drbd1226.out‚0 disk.drbd2118.in‚0* proc.memory.percent‚11.940399805704754 disk.drbd1226.in‚0 disk.drbd1938.in‚0( proc.memory.availableMB‚ 1943.5078125 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0$ memory_percent‚92.09184646606445 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0& disk.nvme0n1.out‚53054.18798828125 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 834.67578125 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 116.953125 network.sent‚ 1.9499826e+07 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 834.67578125 disk.drbd1356.out‚0 proc.cpu.threads‚17 cpu‚25.3368485711659 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0% disk./.usageGB‚1260.8635787963867HÔÊž¢/‚(:“ ü–‹Ó雿³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚293.451' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.81700000000001 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E[ˆž£/‚(:“ ‹—‹Ó™Ä³ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚284.435' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.81166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü¬ùT¤/‚(:É  ‹—‹ÓÄÃæ³ disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚53054.42431640625 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1942.64453125 disk.drbd1931.in‚0* proc.memory.percent‚11.940399805704754 disk.drbd1673.out‚0 disk.sda2.out‚ 834.96484375 disk.drbd1965.in‚0 disk.drbd1356.in‚0 cpu‚25.336808987537957% disk./.usageGB‚1260.8635787963867 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 network.recv‚ 1.461202e+06 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.drbd1550.in‚0 disk.sda.out‚ 834.96484375) disk./.usagePercent‚70.51409188600755 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 116.953125# disk.dm-0.out‚53054.42431640625 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0$ memory_percent‚92.09535916646323# proc.memory.rssMB‚ 2934.47265625 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 116.953125 disk.drbd1510.out‚0 disk.rbd1.out‚101.9404296875 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 network.sent‚ 1.9559872e+07 disk.drbd2111.out‚0g!P©ž¥/‚(:“ š—‹Óúå³ gpu.0.gpu‚71.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚287.311' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.77033333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Z€-O¦/‚(:Ä  š—‹ÓíÔ„´ disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 116.953125 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2934.47265625 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0% proc.memory.availableMB‚ 1942.4375 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 cpu‚25.33676894002743% disk./.usageGB‚1260.8635787963867 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 835.25390625 disk.drbd1356.out‚0 disk.drbd1226.in‚0 network.sent‚ 1.9562917e+07 disk.sda.in‚ 0.03515625 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚53054.58837890625 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0$ memory_percent‚92.09620157877603 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0# disk.dm-0.out‚53054.58837890625 disk.nvme0n1.in‚ 116.953125 disk.drbd1817.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51409188600755 disk.sda.out‚ 835.25390625 disk.drbd2078.out‚0 disk.drbd2111.out‚0* proc.memory.percent‚11.940399805704754 network.recv‚ 1.462297e+06 disk.rbd1.out‚102.0029296875 disk.drbd1285.in‚0m§kï§/‚(:’ ©—‹ÓŠÔº³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚61.0 gpu.0.powerWatts‚109.72' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.57333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û…€D@¨/‚(:µ  ©—‹Ó³îÙ³ disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0 network.recv‚ 1.463274e+06 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚102.0029296875# proc.memory.availableMB‚1942.25 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.dm-0.in‚ 116.953125 disk.drbd1817.in‚0 disk.drbd1289.in‚0" disk.dm-0.out‚53054.9130859375 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 116.953125 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚53054.9130859375 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2934.47265625 disk.sda2.out‚835.75 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 network.sent‚ 1.9565922e+07) disk./.usagePercent‚70.51409145933188 disk.drbd1938.out‚0 disk.drbd1550.out‚0 cpu‚25.336705878800377 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1356.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.sda.out‚835.75* proc.memory.percent‚11.940399805704754% disk./.usageGB‚1260.8635711669922 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1931.in‚0$ memory_percent‚92.09696451822916 disk.drbd1938.in‚0fc„ž©/‚(:“ ¸—‹ÓßîƳ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.899' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.96633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý«î]Oª/‚(:Ä  ¸—‹ÓçÑå³ disk.drbd1552.in‚0 disk.sda1.out‚0% disk./.usageGB‚1260.8635711669922 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0* proc.memory.percent‚11.940399805704754 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1931.in‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.nvme0n1.in‚ 116.953125 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1673.in‚0 cpu‚25.336641802564998 proc.cpu.threads‚17 disk.sda2.out‚ 836.11328125 disk.drbd1285.out‚0 network.sent‚ 1.956951e+07 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 116.953125 disk.rbd1.out‚ 102.03515625 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1938.out‚0" disk.dm-0.out‚53055.3193359375 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.09694862365723 disk.sda1.in‚0 disk.drbd1817.out‚0) proc.memory.availableMB‚ 1942.25390625) disk./.usagePercent‚70.51409145933188 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625# proc.memory.rssMB‚ 2934.47265625 network.recv‚ 1.464186e+06 disk.drbd2078.in‚0% disk.nvme0n1.out‚53055.3193359375 disk.sda.out‚ 836.11328125 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0sòŸ«/‚(:” Ç—‹ÓÂÇ°³ gpu.0.gpu‚76.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.266' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.42200000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—,#J¬/‚(:¿  Ç—‹Ó¶œÒ³ disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51409273935892 disk.drbd1584.out‚0 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 116.953125* proc.memory.percent‚11.940399805704754 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.sda.out‚ 836.6171875 disk.drbd2111.out‚0 disk.drbd1285.out‚0 cpu‚25.336602980696053 disk.drbd1938.in‚0# disk.dm-0.out‚53056.41748046875 disk.drbd1510.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.out‚0$ memory_percent‚92.09696451822916 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2934.47265625 disk.drbd2118.out‚0 disk.dm-0.in‚ 116.953125 disk.drbd1356.out‚0& disk.nvme0n1.out‚53056.41748046875 network.sent‚ 1.9572058e+07 network.recv‚ 1.465075e+06 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 836.6171875 disk.drbd1289.in‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8635940551758 disk.drbd1931.in‚0 disk.rbd1.out‚ 102.12890625 disk.drbd1226.out‚0# proc.memory.availableMB‚1942.25 disk.drbd2111.in‚0 disk.rbd1.in‚0é¢4­// _wandbruntime‚20356  _runtime‚20356C7zÅÈ®/ trainer/global_step‚11199 $ _timestamp‚1.784859593082616e+09 _runtime‚20356.439842377 _step‚931 & lr-AdamW/pg1‚1.163813615017435e-05 & lr-AdamW/pg2‚1.163813615017435e-05<xͯ/Ç trainer/global_step‚11199 $ _timestamp‚1.784859593082616e+09 _runtime‚20356.439842377 _step‚931 & lr-AdamW/pg1‚1.163813615017435e-05 & lr-AdamW/pg2‚1.163813615017435e-05£ëZ3³4°// _wandbruntime‚20356  _runtime‚20356Z“¢r[%±/ŽP O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 $ _timestamp‚1.784859593083666e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚882269474799616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3234134506978364 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11685023438848 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3222467704336816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚896561260767232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚896560835117568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29841491 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3234133136939110 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3223738503784448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14291417876480 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103985053 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3223738503784448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚932 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11685023438848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3223738529957888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 trainer/global_step‚11199 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3222467678908848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103974989 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11686368875008 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103985053 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97202656 _runtime‚20356.441228618 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63966914 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3235424898832896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97202656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63966525 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚882269842890752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34125034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34125418 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14291360317952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11665458030262 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97202662 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pæÄa—¹ool.peak‚ 46288510976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3235423527223296 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3223738529957888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚882269474799616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3234134506978364 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11685023438848 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3222467704336816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚896561260767232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚896560835117568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29841491 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3234133136939110 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3223738503784448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14291417876480 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103985053 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3223738503784448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 _step‚932 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11685023438848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3223738529957888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 trainer/global_step‚11199 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3222467678908848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103974989 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11686368875008 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103985053 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97202656 _runtime‚20356.441228618 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63966914 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3235424898832896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97202656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63966525 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚882269842890752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34125034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34125418 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14291360317952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11665458030262 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97202662 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3235423527223296 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3223738529957888 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29844131 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚896641082435584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11666478831902 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3235704830638592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11687391390208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882348061165568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11667823443300 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201205178 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚20357.915578089 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103984071 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11687391390208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3235703459021824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103994149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3222746480682624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11686045946880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29844136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚896640658890240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚933 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103994149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63972171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3224017439248384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848595945579927e+09Ó7PX$'µ/žN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103984071 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚383 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97211029 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3224017413074944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚882348429256704 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3235704830638592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3234412959514526 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3222746506110592 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3224017413074944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201205178 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 _runtime‚20357.915578089 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103984071 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11687391390208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3235703459021824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103994149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3222746480682624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11686045946880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29844136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚896640658890240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚933 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103994149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63972171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3224017439248384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 % _timestamp‚1.7848595945579927e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63972554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34128418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚378 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29844131 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚896641082435584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11666478831902 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3235704830638592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11687391390208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882348061165568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11667823443300 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 B&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 particle/accuracy_2_of_3‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7971538305282593 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6277579069137573 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2899337708950043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6875087022781372 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22165438532829285 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9651569724082947 6validation_average_jet_accuracy‚0.4016757309436798 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.641574501991272 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17158521711826324 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4506992995738983 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8441557884216309 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 particle/accuracy_1_of_1‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03611398860812187 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.34999486804008484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 +particle/sensitivity‚0.6369865536689758 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2743300795555115 'particle/f_score‚0.6518154740333557 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6621206998825073 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17733371257781982 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4628472626209259 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7343496084213257 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.714624285697937 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0517217181622982 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36210867762565613 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6031442284584045 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3043917119503021 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6746083498001099 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8729758262634277 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3604355752468109 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_1_of_2‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8430094122886658 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3839762508869171 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2860969603061676 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9215311408042908 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6594598889350891 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297714114189148 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 *jet/accuracy_1_of_4‚0.9832039475440979 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7164067029953003 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1599932610988617 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4640224874019623 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7664524912834167 particle/accuracy_2_of_2‚1 _runtime‚20377.338181723 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028817366808652878 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6389041543006897 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7094934582710266 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08636099100112915 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38265880942344666 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30237773060798645 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2640071213245392 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04261501878499985 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.750354895611255 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 trainer/global_step‚11199 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6426949501037598 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5434644818305969 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7854025959968567 )jet/accuracy_2_of_4‚0.842309832572937 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7989381551742554 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06993712484836578 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3497717082500458 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7744528651237488 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5016787052154541 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016900094226002693 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 _step‚935 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7696470618247986 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8832986950874329 % _timestamp‚1.7848596139799342e+09 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8452072143554688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490260243415833 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3482796549797058 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7447789311408997 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6845528483390808 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3815213143825531 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.797825038433075 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8832986950874329 )jet/accuracy_2_of_3‚0.603666365146637 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3985803425312042 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6369866728782654 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2656966745853424 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6307691931724548 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9486009478569031 *jet/accuracy_2_of_2‚0.1700398474931717 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7093101143836975 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3396984040737152 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8216089606285095 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8038768768310547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1721445620059967 *jet/accuracy_1_of_1‚0.3671167194843292 epoch‚39 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8778425455093384 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03540017083287239 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.37845250964164734 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7047038674354553 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2844006419181824 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07876145094633102 +particle/specificity‚0.9132838249206543 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18939568102359772 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6999751925468445 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3485929071903229 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7517818212509155 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29288944602012634 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6273823976516724 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21821488440036774 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6550878882408142 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 )jet/accuracy_1_of_3‚0.934964656829834 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.35975345969200134 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7069640159606934 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_4‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7155030369758606 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6871317625045776 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016900094226002693 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5115094780921936 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6507276296615601 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5774000287055969 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1403288096189499 +jet/accuracy_3_of_4‚0.11347367614507675 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5935816168785095 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6426949501037598 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3482796549797058 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032é}ó1¯:½/©u =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 particle/accuracy_1_of_2‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8430094122886658 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3839762508869171 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2860969603061676 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9215311408042908 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6594598889350891 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297714114189148 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 *jet/accuracy_1_of_4‚0.9832039475440979 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7164067029953003 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1599932610988617 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4640224874019623 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7664524912834167 particle/accuracy_2_of_2‚1 _runtime‚20377.338181723 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028817366808652878 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6389041543006897 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7094934582710266 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08636099100112915 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38265880942344666 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30237773060798645 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2640071213245392 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04261501878499985 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.750354895611255 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 trainer/global_step‚11199 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6426949501037598 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5434644818305969 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7854025959968567 )jet/accuracy_2_of_4‚0.842309832572937 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7989381551742554 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06993712484836578 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3497717082500458 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7744528651237488 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5016787052154541 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016900094226002693 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 _step‚935 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7696470618247986 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8832986950874329 % _timestamp‚1.7848596139799342e+09 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8452072143554688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490260243415833 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3482796549797058 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7447789311408997 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6845528483390808 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3815213143825531 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.797825038433075 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8832986950874329 )jet/accuracy_2_of_3‚0.603666365146637 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3985803425312042 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6369866728782654 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2656966745853424 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6307691931724548 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9486009478569031 *jet/accuracy_2_of_2‚0.1700398474931717 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7093101143836975 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3396984040737152 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8216089606285095 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8038768768310547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1721445620059967 *jet/accuracy_1_of_1‚0.3671167194843292 epoch‚39 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8778425455093384 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03540017083287239 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.37845250964164734 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7047038674354553 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2844006419181824 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07876145094633102 +particle/specificity‚0.9132838249206543 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18939568102359772 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6999751925468445 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3485929071903229 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7517818212509155 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29288944602012634 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6273823976516724 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21821488440036774 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6550878882408142 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 )jet/accuracy_1_of_3‚0.934964656829834 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.35975345969200134 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7069640159606934 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_4‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7155030369758606 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6871317625045776 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016900094226002693 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5115094780921936 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6507276296615601 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5774000287055969 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1403288096189499 +jet/accuracy_3_of_4‚0.11347367614507675 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5935816168785095 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6426949501037598 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3482796549797058 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27403023838996887 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6883414387702942 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2819870710372925 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8540039658546448 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6605652570724487 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02828500233590603 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 particle/accuracy_2_of_3‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7971538305282593 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6277579069137573 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2899337708950043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6875087022781372 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22165438532829285 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9651569724082947 6validation_average_jet_accuracy‚0.4016757309436798 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.641574501991272 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17158521711826324 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4506992995738983 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8441557884216309 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 particle/accuracy_1_of_1‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03611398860812187 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.34999486804008484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7769231200218201 +particle/sensitivity‚0.6369865536689758 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2743300795555115 'particle/f_score‚0.6518154740333557 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6621206998825073 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17733371257781982 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4628472626209259 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7343496084213257 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.714624285697937 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0517217181622982 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36210867762565613 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6031442284584045 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3043917119503021 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6746083498001099 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8729758262634277 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3604355752468109 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718§ .…¹ž¾/‚(:“ å—‹Óô¹³ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚296.155' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.71833333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í?¤ç¿/‚(:¾  å—‹Óضӳ disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.dm-0.out‚53057.44921875 network.recv‚ 1.46828e+06 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1584.iìèa’bn‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1936.86328125 disk.drbd1550.in‚0 disk.dm-0.in‚ 116.953125 disk.drbd2078.in‚0 disk.drbd1356.in‚0 cpu‚25.338887731567226% disk./.usageGB‚1260.8635330200195 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 116.953125 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0$ memory_percent‚92.11888313293457 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 102.55078125# disk.nvme0n1.out‚53057.44921875 disk.drbd1817.out‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2939.73828125 network.sent‚ 1.9635679e+07 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 837.2734375 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 837.2734375 disk.drbd1289.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.961825688680012) disk./.usagePercent‚70.51408932595345 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0aœåžÀ/‚(:“ ô—‹ÓÐÃɳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚285.309' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.10300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j*X=FÁ/‚(:»  ô—‹ÓÌŒî³ disk.drbd1965.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 116.953125 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 116.953125 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0$ memory_percent‚92.11934407552084 disk.drbd2041.in‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8635330200195# disk.nvme0n1.out‚53057.67578125 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2939.73828125 network.recv‚ 1.470262e+06 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.dm-0.out‚53057.67578125 disk.sda2.out‚ 837.58203125 disk.drbd2111.out‚0 disk.sda.in‚ 0.03515625 cpu‚25.338821935154783 disk.drbd1938.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51408932595345 disk.drbd1817.out‚0# proc.memory.availableMB‚1936.75* proc.memory.percent‚11.961825688680012 network.sent‚ 1.9689571e+07 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 837.58203125 disk.rbd1.out‚ 102.61328125 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0UU§žÂ/‚(:“ ƒ˜‹ÓÐÀÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚266.785' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.92833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý ¥ÈOÃ/‚(:Ä  ƒ˜‹Ó¤»ü³ disk.drbd1938.out‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2939.73828125% disk./.usageGB‚1260.8654861450195 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 838.01953125 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 116.953125$ memory_percent‚92.11945533752441 cpu‚25.33878093819872 network.recv‚ 1.471261e+06 disk.drbd1356.out‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.961825688680012 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 102.70703125 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0 network.sent‚ 1.9692028e+07 disk.drbd1289.out‚0 disk.sda.out‚ 838.01953125 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0) proc.memory.availableMB‚ 1936.72265625 disk.dm-0.in‚ 116.953125 disk.drbd2111.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0) disk./.usagePercent‚70.51419855492797 disk.drbd1552.out‚0 disk.drbd2118.in‚0" disk.dm-0.out‚53058.1533203125 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚53058.1533203125 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0ÁGçúŸÄ/‚(:” ’˜‹ÓÝݳ gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚304.762' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.58733333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙèkNÅ/‚(:à  ’˜‹Ó•Ëü³) proc.memory.availableMB‚ 1936.69140625 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51419855492797% disk./.usageGB‚1260.8654861450195 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 network.sent‚ 1.9695066e+07 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.sda.out‚ 838.3984375 disk.sda2.out‚ 838.3984375 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0$ memory_percent‚92.11958249409994# proc.memory.rssMB‚ 2939.73828125 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 116.953125 disk.drbd1584.out‚0 cpu‚25.338728075110748 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0* proc.memory.percent‚11.961825688680012 disk.dm-0.in‚ 116.953125 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.rbd1.out‚ 102.76953125% disk.nvme0n1.out‚53058.3330078125 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1226.out‚0" disk.dm-0.out‚53058.3330078125 network.recv‚ 1.472151e+06 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0ÆdÿžÆ/‚(:“ ¡˜‹Ó“ù׳ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚291.628' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.20933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ›MÇ/‚(:  ¡˜‹Óñç÷³ disk.drbd2118.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.961825688680012 disk.drbd1828.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 102.76953125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.sda.out‚ 838.68359375 disk.drbd2078.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1356.out‚0 network.sent‚ 1.9698066e+07) disk./.usagePercent‚70.51419855492797 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8654861450195 disk.rbd1.in‚0 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2939.73828125 disk.sda1.out‚0 disk.drbd1289.in‚0# disk.dm-0.out‚53058.72119140625 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0$ memory_percent‚92.11959838867188 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 838.68359375 disk.drbd1938.out‚0 cpu‚25.33866318080372 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0% proc.memory.availableMB‚ 1936.6875 network.recv‚ 1.473192e+06 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 116.953125 disk.nvme0n1.in‚ 116.953125& disk.nvme0n1.out‚53058.72119140625Eº¼È/‚(:’ °˜‹Óí§Ø³ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚294.26' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.08666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07$ÌßOÉ/‚(:Ä  °˜‹Ó½Ýõ³ network.sent‚ 1.9701109e+07 network.recv‚ 1.47421e+06 disk.drbd1552.in‚0 disk.drbd1931.out‚0$ memory_percent‚92.11959838867188 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0% proc.memory.availableMB‚ 1936.6875% disk./.usageGB‚1260.8655052185059 disk.sda1.in‚0* proc.memory.percent‚11.961825688680012 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2939.73828125 cpu‚25.338634923549257 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 116.953125 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.sda.in‚ 0.03515625# disk.dm-0.out‚53058.88916015625 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 839.02734375 disk.drbd1828.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51419962161718 disk.drbd2118.out‚0 disk.dm-0.in‚ 116.953125 disk.drbd2078.in‚0& disk.nvme0n1.out‚53058.88916015625 disk.drbd1817.in‚0 disk.sda.out‚ 839.02734375 disk.drbd2111.in‚0 disk.rbd1.out‚102.8017578125 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1828.in‚0º†4Ê// _wandbruntime‚20462  _runtime‚20462æ‘î}ËË/Å ' lr-AdamW/pg1‚1.1240062865143476e-05 ' lr-AdamW/pg2‚1.1240062865143476e-05 trainer/global_step‚11249 % _timestamp‚1.7848596990818834e+09 _runtime‚20462.439155447 _step‚936ÉWD­ÐÌ/Ê ' lr-AdamW/pg1‚1.1240062865143476e-05 ' lr-AdamW/pg2‚1.1240062865143476e-05 trainer/global_step‚11249 % _timestamp‚1.7848596990818834e+09 _runtime‚20462.439155447 _step‚936¨€]­4Í// _wandbruntime‚20462  _runtime‚204629V m(Î/P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚900609180800512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3251986476465664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚886247792610304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3240244671207424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848596990830178e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11743176867840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3240244645033984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚900609595964416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11722170752379 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202188686 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97710438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202198756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3240244645033984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104488312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14361435262976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104488312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11743176867840 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3251986476465664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64279860 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3251987848075264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14361388190208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 _runtime‚20462.440505845 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11741831431680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34299477 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚937 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97710438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11741831431680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97710444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104478248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34299814 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚937 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97710438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11741831431680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97710444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104478248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34299814 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3251986476465664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚886247792610304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3240244671207424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848596990830178e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11743176867840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3240244645033984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚900609595964416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11722170752379 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202188686 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97710438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202198756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3240244645033984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104488312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14361435262976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104488312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11743176867840 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3251986476465664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64279860 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3251987848075264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14361388190208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 _runtime‚20462.440505845 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11741831431680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34299477 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11724536165417 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11742853939712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104487330 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚11249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3252267779880960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886326378976256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886326747067392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3252267779880960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34302730 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚900689012429312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14362686810112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3240523580497920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 % _timestamp‚1.7848597005578558e+09 _step‚938 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3239245570522592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11744199383040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202216219 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3240523554324480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104497408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64285758 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34302396 _runtime‚20463.915439725 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202206135 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3250970132115977 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29983028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29983023 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11723191554019 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97718805 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202216219 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3252266408264192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚334 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104487330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3239245595950560 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11742853939712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64285419 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864yhsC$'Ò/žN FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202216219 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3240523554324480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104497408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64285758 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34302396 _runtime‚20463.915439725 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202206135 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3250970132115977 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29983028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29983023 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11723191554019 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97718805 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202216219 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3252266408264192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚334 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104487330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3239245595950560 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11742853939712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64285419 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3240523580497920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104497408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11724536165417 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11742853939712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104487330 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚11249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3252267779880960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886326378976256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886326747067392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3252267779880960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34302730 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚900689012429312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14362686810112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3240523580497920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 % _timestamp‚1.7848597005578558e+09 _step‚938 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3239245570522592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11744199383040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3265983130546688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14423863732224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3265984502156288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚890177110907904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203062988 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3254190109558784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203073058 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104944264 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3252906857437616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3265983130546688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚904600974640128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚904601391901184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _step‚941 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98128788 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3254190109558784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11793020987904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64558517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30112383 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34446129 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3264681502112057 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104934200 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98128794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3265983130546688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14423863732224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3265984502156288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚890177110907904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203062988 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3254190109558784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203073058 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104944264 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3252906857437616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3265983130546688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚353 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚904600974640128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚904601391901184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 _step‚941 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98128788 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3254190109558784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11793020987904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64558517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30112383 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34446129 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3264681502112057 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104934200 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005­c¯Ò™“æ/‚(:ˆ Š™‹Ó”¶¯³ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚294.678' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÕ¤kGç/‚(:¼  Š™‹ÓÒèȳ& disk.nvme0n1.out‚53061.45068359375 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚117.125 disk.sda2.out‚ 841.046875 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 network.sent‚ 1.9776692e+07 disk.drbd2111.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0( proc.memory.availableMB‚ 1937.3828125 disk.drbd1584.in‚0$ memory_percent‚92.11676915486655 disk.drbd1285.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 841.046875 disk.drbd1931.in‚0 network.recv‚ 1.480938e+06 disk.dm-0.in‚117.125 disk.drbd1356.in‚0# disk.dm-0.out‚53061.45068359375 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 proc.cpu.threads‚17$ disk./.usageGB‚1260.864372253418 disk.rbd1.in‚0) disk./.usagePercent‚70.51413626027845 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.03515625# proc.memory.rssMB‚ 2939.73828125 cpu‚25.338309013077044 disk.rbd1.out‚ 103.248046875 disk.sda1.in‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.961825688680012 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd1817.in‚0qLà4è// _wandbruntime‚20550  _runtime‚20550ÈÚ•2'é/˜N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _step‚942 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3253185684639360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3253185659211392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14425172985344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11774295436827 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98137155 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3254469045022720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104943282 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3266264433961984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64563736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚890255697273856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104953360 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203080437 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98137161 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3266264433961984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11795388939264 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11775640048225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 $ _timestamp‚1.784859787305418e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚358 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚904680816902144 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3266263062345216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11794043495936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203080437 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚890256065364992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3254469018849280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14425119628288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104953360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3264959954648219 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203090521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30115023 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚20550.662998479 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3254469045022720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203090521 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353ûýãóê/N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203090521 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353 UEDeviceStatsMonitor.o'¶,·%n_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _step‚942 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3253185684639360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3253185659211392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14425172985344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11774295436827 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98137155 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3254469045022720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104943282 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3266264433961984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64563736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚890255697273856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104953360 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203080437 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98137161 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3266264433961984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11795388939264 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11775640048225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 $ _timestamp‚1.784859787305418e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚358 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚904680816902144 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3266263062345216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11794043495936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203080437 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚890256065364992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3254469018849280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14425119628288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104953360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3264959954648219 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203090521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30115023 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚20550.662998479 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3254469045022720®îc~e4ë// _wandbruntime‚20550  _runtime‚20550@vQ ì/‡ 3loss/SRqqt1/assignment_loss‚0.48543863120514724 /loss/FRt1/detection_loss‚0.4941745932474569 /loss/FBt2/detection_loss‚0.4512905988568357 2loss/SRqqt2/assignment_loss‚0.5002095834263439 /loss/FBt1/detection_loss‚0.3698948869843864 %loss/total_loss‚7.648342854142582 _runtime‚20550.703952428 2loss/FBt2/assignment_loss‚0.008793972138566497 .loss/FRt2/detection_loss‚1.417562680258933 1loss/SRqqt1/detection_loss‚0.9308127571355782 1loss/SRqqt2/detection_loss‚0.6072228163254618 trainer/global_step‚11299 % _timestamp‚1.7848597873467495e+09 _step‚943 0loss/FRt1/assignment_loss‚1.6373624655307752 0loss/FRt2/assignment_loss‚0.7432838062445323 3loss/FBt1/assignment_loss‚0.0022960627885648677 epoch‚40Ӣܛí/Œ 0loss/FRt1/assignment_loss‚1.6373624655307752 0loss/FRt2/assignment_loss‚0.7432838062445323 3loss/FBt1/assignment_loss‚0.0022960627885648677 epoch‚40 3loss/SRqqt1/assignment_loss‚0.48543863120514724 /loss/FRt1/detection_loss‚0.4941745932474569 /loss/FBt2/detection_loss‚0.4512905988568357 2loss/SRqqt2/assignment_loss‚0.5002095834263439 /loss/FBt1/detection_loss‚0.3698948869843864 %loss/total_loss‚7.648342854142582 _runtime‚20550.703952428 2loss/FBt2/assignment_loss‚0.008793972138566497 .loss/FRt2/detection_loss‚1.417562680258933 1loss/SRqqt1/detection_loss‚0.9308127571355782 1loss/SRqqt2/detection_loss‚0.6072228163254618 trainer/global_step‚11299 % _timestamp‚1.7848597873467495e+09 _step‚943¯ 1Gî/‚(:¼  ™™‹Ó¨‚ˆ´ disk.sda.in‚ 0.03515625 disk.sda.out‚ 841.42578125 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 proc.cpu.threads‚17 network.sent‚ 1.982588e+07$ disk./.usageGB‚1260.864372253418 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚117.125 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.961825688680012 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1937.578125# proc.memory.rssMB‚ 2939.73828125 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.sda2.in‚ 0.03515625$ memory_percent‚92.11597442626953) disk./.usagePercent‚70.51413626027845 disk.drbd2111.in‚0 disk.drbd2111.out‚0 cpu‚25.338267703384542 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚117.125 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0% disk.nvme0n1.out‚53061.9619140625 disk.drbd1550.out‚0 disk.sda2.out‚ 841.42578125 network.recv‚ 1.482471e+06 disk.drbd1289.out‚0" disk.dm-0.out‚53061.9619140625 disk.rbd1.out‚ 103.248046875 disk.drbd1226.out‚0 disk.drbd1510.in‚0!Wõ%žï/‚(:“ ™™‹Ó­Õç³ gpu.0.gpu‚42.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚296.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.97566666666667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê™uyŸð/‚(:” ¨™‹Óïô²³ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.128' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.70933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~îàJñ/‚(:¿  ¨™‹Óøѳ disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚117.125 disk.sda1.out‚0" disk.dm-0.out‚53062.1728515625 disk.drbd1938.out‚0* proc.memory.percent‚11.961825688680012 network.recv‚ 1.484365e+06 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda1.in‚0) disk./.usagePercent‚70.51392825588361 disk.drbd2111.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 841.66015625 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.sda2.in‚ 0.03515625 disk.sda.out‚ 841.66015625$ disk./.usageGB‚1260.860652923584 disk.drbd1931.in‚0 cpu‚25.338204588406427 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2939.73828125 disk.drbd1965.in‚0# memory_percent‚92.1159585316976 disk.rbd1.out‚103.2802734375 disk.drbd1584.out‚0 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1937.58203125 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚117.125 disk.drbd1965.out‚0 disk.drbd2078.in‚0 network.sent‚ 1.9874849e+07% disk.nvme0n1.out‚53062.1728515625‰g±Kò/‚(:À  ·™‹Óî«Ö³ disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0$ disk.nvme0n1.out‚53062.876953125 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.sda.in‚ 0.03515625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 119.03125) proc.memory.availableMB‚ 1937.33203125 cpu‚25.33816457693134 disk.drbd1828.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0$ memory_percent‚92.11697578430176* proc.memory.percent‚11.961825688680012 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 842.09765625 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.8607177734375 network.recv‚ 1.485374e+06 disk.drbd1285.out‚0! disk.dm-0.out‚53062.876953125 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.dm-0.in‚ 119.03125# proc.memory.rssMB‚ 2939.73828125( disk./.usagePercent‚70.5139318826269 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 842.09765625 disk.drbd2078.out‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 network.sent‚ 1.987843e+07 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.rbd1.out‚103.3818359375 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625+½šèžó/‚(:“ ·™‹Óµ³ gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚274.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.33366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ï ±Ÿô/‚(:” Æ™‹Óչೠgpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚63.0 gpu.0.powerWatts‚308.705' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.90166666666667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÚЬMõ/‚(:  Æ™‹Ó¼¯ý³ disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.rbd1.out‚103.4443359375 disk.drbd1938.out‚0 cpu‚25.338123945676426 disk.nvme0n1.in‚ 119.03125 disk.sda.out‚ 842.34765625 disk.sda2.out‚ 842.34765625 disk.drbd1965.out‚0 disk.drbd2078.in‚0$ memory_percent‚92.11697578430176 disk.drbd1356.in‚0$ disk.nvme0n1.out‚53063.060546875 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8607177734375 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1937.33203125 disk.dm-0.in‚ 119.03125 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0( disk./.usagePercent‚70.5139318826269 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2939.73828125 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.03515625* proc.memory.percent‚11.961825688680012 disk.drbd1673.out‚0 disk.drbd1965.in‚0 network.sent‚ 1.9880715e+07 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 network.recv‚ 1.485572e+06 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0! disk.dm-0.out‚53063.060546875 disk.drbd1584.in‚0 disk.drbd1356.out‚0Q_\qŸö/‚(:” Õ™‹Ó¹ª³ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.967300415039055* gpu.0.memoryAllocatedBytes‚ 49793531904 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.345' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.44833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤€¥‹O÷/‚(:Ä  Õ™‹Óñóʳ disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0 disk.sda.out‚ 842.8359375 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2939.73828125 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625# disk.dm-0.out‚53063.54345703125 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 network.recv‚ 1.486733e+06 disk.sda2.out‚ 842.8359375 disk.drbd1550.out‚0% disk./.usageGB‚1260.8606796264648 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1285.in‚0 cpu‚25.338060978911855 network.sent‚ 1.9886345e+07 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.961825688680012( disk./.usagePercent‚70.5139297492485 disk.rbd1.out‚103.4443359375$ memory_percent‚92.11780230204263 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 119.03125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚53063.54345703125 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1937.12890625 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0)yùX4ø// _wandbruntime‚20635  _runtime‚20635P{,Êù/Ä ' lr-AdamW/pg1‚1.0462577113163311e-05 ' lr-AdamW/pg2‚1.0462577113163311e-05 trainer/global_step‚11349 $ _timestamp‚1.784859872508715e+09 _runtime‚20635.865949915 _step‚944˜ï„ÏÏú/É ' lr-AdamW/pg2‚1.0462577113163311e-05 trainer/global_step‚11349 $ _timestamp‚1.784859872508715e+09 _runtime‚20635.865949915 _step‚944 ' lr-AdamW/pg1‚1.0462577113163311e-05°ÝkËÝ4û// _wandbruntime‚20635  _runtime‚20635»Î(ü/ŽP BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3279979784627712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11845555980288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64837835 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30244383 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14485956266496 _step‚945 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚894106797296640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98547138 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3268135600257024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11844210544128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98547138 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11825723129281 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105390152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚894106429205504 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3279981156237312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848598725097349e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚908592815315968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3279981156237312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105400216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30244388 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3266846971554384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968(êÞ\ý/“P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11844210544128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98547138 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11825723129281 F=DeviceStRzx½½%atsMonitor.on_train_batch_start/segment.all.allocated‚2005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105390152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚894106429205504 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3279981156237312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7848598725097349e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚908592815315968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3279981156237312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105400216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30244388 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3266846971554384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚375 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3279979784627712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11845555980288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64837835 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30244383 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14485956266496 _step‚945 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚894106797296640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98547138 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3268135600257024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚370 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1728053248±Ñ:&ƒ4þ// _wandbruntime‚20637  _runtime‚20637ð»–—'ÿ/˜N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3267125773328160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _step‚946 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11845233052160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚894185015571456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14487253976064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3278952517259193 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3278951147219827 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34596352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203964823 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚908672637638656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3280261088043008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203954739 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98555511 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105399234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30247028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11825399319635 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98555511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3268414483374080 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11846578495488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚364 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34595993 $ _timestamp‚1.784859873986325e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64843016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚11349 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98555505 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3280261088043008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚894185383662592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98555505 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30247023 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64843380 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904YЯ™#'€0N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203954739 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3268414509547520 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64843016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 trainer/global_step‚11349 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98555505 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3280261088043008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚894185383662592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98555505 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30247023 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64843380 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3267125773328160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 _step‚946 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11845233052160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚894185015571456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14487253976064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3278952517259193 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3278951147219827 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34596352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203964823 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚908672637638656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3280261088043008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203954739 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98555511 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105399234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30247028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11825399319635 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98555511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3268414483374080 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11846578495488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚364 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34595993 $ _timestamp‚1.784859873986325e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3293976438708736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14548121101824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105856168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3282081038608384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚912584236696064 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚20722.631875007 _step‚949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3282081038608384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14548061446144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65116529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65116142 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚11399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98965494 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11895400100352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11896745536512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11895400100352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3293976438708736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14548121101824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105856168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3282081038608384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚912584236696064 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚20722.631875007 _step‚949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3282081038608384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14548061446144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65116529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65116142 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49243226112 trainer/global_step‚11399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98965494 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11895400100352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11896745536512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11895400100352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49243226112 J4–0/ _wandbruntime‚20724  _runtime‚20724Äû—0™N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚378 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚898114333869056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105865264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚912663632320512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14549360197120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11877847813841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11897768051712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204839125 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11876503202443 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105855186 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204839125 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14549298451456 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11897768051712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30379028 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3294256370507264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3282359947898880 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3282359974072320 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3282359947898880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98973861 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65121760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105855186 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11896422608384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 % _timestamp‚1.7848599607480185e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204829041 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204829041 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚20724.105599703 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328>×é$'˜0žN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3282359974072320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98973855 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3294257742124032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3282359974072320 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3282359947898880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98973861 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65121760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105855186 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11896422608384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2005 % _timestamp‚1.7848599607480185e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204829041 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204829041 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚20724.105599703 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚378 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚898114333869056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105865264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚912663632320512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14549360197120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49243226112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚627 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11877847813841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11897768051712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204839125 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11876503202443 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105855186 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204839125 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14549298451456 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11897768051712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30379028 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3294256370507264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3282359947898880 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3296026503133184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚11449 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205695964 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99383844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99383844 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚901965433891840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30508383 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106302056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205685894 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99383838 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3307974464399360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3307973092789760 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3296026503133184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99383838 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6ÔãÌ(­0”P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99383844 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚901965433891840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30508383 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106302056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205685894 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99383838 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3307974464399360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3307973092789760 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3296026503133184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99383838 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1728053248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65396227 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚659 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11927930894897 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11946589656576 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106302056 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3296026529306624 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚382 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3306655079826881 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49243226112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106312120 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3307973092789760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11947935092736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11947935092736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3296026503133184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1728053248 trainer/global_step‚11449 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205695964 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99383844 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11947612164608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34891146 _runtime‚20810.74087551 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99392211 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚380 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3295005925277792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99392205 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3308253024588288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11947612164608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205713427 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205703343 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106321216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3295005950705760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11927607085251 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3296305438597120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99392205 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3306933532363043 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848600473832753e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚375 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11948957607936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30511028 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚902044020257792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3296305438597120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106311138 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65402174 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3296305412423680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106311138 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65401794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34890771 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚916655249561088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14611597394432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205703343 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11948957607936$4•Þ°0N SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1728053248 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99392211 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚380 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.larŒ .E"ge_pool.freed‚3295005925277792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1728053248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99392205 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3308253024588288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11947612164608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205713427 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205703343 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49243226112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106321216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3295005950705760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚659 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11927607085251 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3296305438597120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99392205 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3306933532363043 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚824 % _timestamp‚1.7848600473832753e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚375 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11948957607936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30511028 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚902044020257792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2005 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3296305438597120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106311138 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65402174 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3296305412423680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1728053248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106311138 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65401794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34890771 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚916655249561088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14611597394432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205703343 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11948957607936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2005 trainer/global_step‚11449 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3308254396205056 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49243226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11947612164608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34891146 _runtime‚20810.74087551 F&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3527836799621582 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_1_of_2‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.630410373210907 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8001251816749573 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28237953782081604 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.16608068346977234 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6166922450065613 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7800468802452087 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9671996831893921 _runtime‚20882.247025513 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚956 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13466010987758636 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36776304244995117 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6063402891159058 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5132553577423096 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5473074316978455 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8452072143554688 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38172250986099243 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7156171798706055 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0667744055390358 particle/accuracy_2_of_4‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025877149775624275 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36638912558555603 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_3‚0.07736089080572128 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7204455137252808 6validation_average_jet_accuracy‚0.4040999710559845 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29300346970558167 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 *jet/accuracy_2_of_4‚0.8421397805213928 +particle/specificity‚0.9128689765930176 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 'particle/f_score‚0.6543490290641785 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9222854971885681 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4592687487602234 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15581963956356049 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6643744707107544 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 *jet/accuracy_1_of_4‚0.9794445633888245 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26831918954849243 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 *jet/accuracy_1_of_2‚0.7895225882530212 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7502743005752563 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3810007870197296 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8762837648391724 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7067688703536987 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36563044786453247 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *loss/val_total_loss‚7.7400257177999965 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87498539686203 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8051037192344666 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7537375688552856 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2962358593940735 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27551689743995667 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17384089529514313 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6661224365234375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6369920372962952 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7992817163467407 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_4‚0.11347367614507675 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03386513888835907 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26701030135154724 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚11479 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6389027833938599 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3053908050060272 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6905365586280823 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 *jet/accuracy_2_of_3‚0.6046562194824219 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.945624828338623 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8352271914482117 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6941428184509277 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02530379593372345 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3543403744697571 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7036469578742981 (particle/accuracy‚0.8546258211135864 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8430094122886658 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051500141620635986 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2268570214509964 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.710346519947052 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.76937335729599 9&Purity/0FRt*SRqqt*FBt/event_p•Nųn roportion‚0.277587890625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16838160157203674 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8832987546920776 particle/accuracy_1_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40350645780563354 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46834179759025574 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6303841471672058 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6211538314819336 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7998250722885132 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3544035851955414 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6876626014709473 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3838655948638916 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5235010981559753 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6326540112495422 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6420146226882935 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27820608019828796 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7307738065719604 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3697091042995453 ,jet/accuracy_4_of_4‚0.018756305798888206 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08322541415691376 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5908083915710449 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3855883479118347 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34520071744918823 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6629926562309265 +jet/accuracy_2_of_2‚0.17135301232337952 +particle/sensitivity‚0.6413959860801697 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3544375002384186 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7191829085350037 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6934621930122375 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6567177176475525 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.307741641998291 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 % _timestamp‚1.7848601188889349e+09 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8126803636550903 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28544968366622925 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚40 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764ƒ¹‡F¸:À0²u 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8430094122886658 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051500141620635986 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2268570214509964 ,"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.710346519947052 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.76937335729599 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16838160157203674 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9960498213768005 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8832987546920776 particle/accuracy_1_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40350645780563354 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9943877458572388 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46834179759025574 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6303841471672058 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6211538314819336 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7998250722885132 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3544035851955414 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6876626014709473 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3838655948638916 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5235010981559753 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6326540112495422 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6420146226882935 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27820608019828796 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7307738065719604 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3697091042995453 ,jet/accuracy_4_of_4‚0.018756305798888206 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08322541415691376 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5908083915710449 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3855883479118347 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34520071744918823 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6629926562309265 +jet/accuracy_2_of_2‚0.17135301232337952 +particle/sensitivity‚0.6413959860801697 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3544375002384186 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7191829085350037 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6934621930122375 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6567177176475525 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.307741641998291 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 % _timestamp‚1.7848601188889349e+09 , Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8126803636550903 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28544968366622925 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚40 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7219020128250122 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490260243415833 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6527776718139648 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.042362142354249954 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7751464247703552 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.64799565076828 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18625745177268982 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6756072044372559 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6420146226882935 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8832987546920776 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2233290672302246 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7207325100898743 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034666530787944794 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28680160641670227 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 *jet/accuracy_1_of_3‚0.9355612993240356 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3543403744697571 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4695170223712921 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3527836799621582 ,"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_1_of_2‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.630410373210907 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8001251816749573 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28237953782081604 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.16608068346977234 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6166922450065613 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7800468802452087 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975429177284241 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9671996831893921 _runtime‚20882.247025513 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 -"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0625 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965167045593262 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚956 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13466010987758636 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36776304244995117 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6063402891159058 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5132553577423096 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5473074316978455 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8452072143554688 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38172250986099243 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7156171798706055 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0667744055390358 particle/accuracy_2_of_4‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025877149775624275 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36638912558555603 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 +jet/accuracy_3_of_3‚0.07736089080572128 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7204455137252808 6validation_average_jet_accuracy‚0.4040999710559845 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29300346970558167 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.012418300844728947 *jet/accuracy_2_of_4‚0.8421397805213928 +particle/specificity‚0.9128689765930176 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 'particle/f_score‚0.6543490290641785 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9222854971885681 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4592687487602234 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15581963956356049 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6643744707107544 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.993851363658905 *jet/accuracy_1_of_4‚0.9794445633888245 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 -"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26831918954849243 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 *jet/accuracy_1_of_2‚0.7895225882530212 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7502743005752563 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3810007870197296 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8762837648391724 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7067688703536987 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36563044786453247 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *loss/val_total_loss‚7.7400257177999965 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.998287558555603 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87498539686203 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8051037192344666 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 + Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9375 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7537375688552856 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2962358593940735 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27551689743995667 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17384089529514313 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6661224365234375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6369920372962952 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7992817163467407 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_4‚0.11347367614507675 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03386513888835907 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26701030135154724 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚11479 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6389027833938599 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3053908050060272 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6905365586280823 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.012418300844728947 *jet/accuracy_2_of_3‚0.6046562194824219 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.945624828338623 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8352271914482117 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6941428184509277 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02530379593372345 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3543403744697571 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7036469578742981 (particle/accuracy‚0.8546258211135864¼cvãDÁ0‚(:¹  㛋ÓíÈî³ disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51426319629374 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.out‚0 network.sent‚ 2.0129293e+07 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1552.in‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd1584.out‚0 disk.drbd1817.out‚0$ disk./.usageGB‚1260.866641998291 disk.sda2.out‚849.1875 disk.dm-0.in‚ 119.03125 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 network.recv‚ 1.500546e+06 disk.drbd1938.in‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚53070.35791015625 disk.drbd1584.in‚0 disk.drbd1817.in‚0% proc.memory.availableMB‚ 1934.3125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 119.03125 disk.sda.out‚849.1875 disk.drbd1931.out‚0 cpu‚25.33952829333915 proc.cpu.threads‚17# disk.dm-0.out‚53070.35791015625 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0$ memory_percent‚92.12926228841147* proc.memory.percent‚11.972570419311523 disk.rbd1.out‚104.7529296875 ñœÂ0‚(:‘ 㛋ÓëÌγ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚308.655' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.88499999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p­Z`Ã0‚(:’ ò›‹Óˆºæ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚282.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.30900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®RJPÄ0‚(:Å  ò›‹Óþ»‰´ disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚53070.69482421875) disk./.usagePercent‚70.51426319629374 disk.drbd1550.out‚0 disk.sda1.in‚0$ disk./.usageGB‚1260.866641998291 disk.drbd1550.in‚0 disk.drbd1584.out‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.sda.out‚ 849.56640625# memory_percent‚92.1302318572998* proc.memory.percent‚11.972570419311523 disk.rbd1.out‚104.8154296875# disk.dm-0.out‚53070.69482421875 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 849.56640625 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1934.07421875 disk.drbd1938.out‚0 disk.drbd1510.in‚0 cpu‚25.339463632014372 network.sent‚ 2.0131745e+07 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.03515625 network.recv‚ 1.501309e+06 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0×MJ4Å0/ _wandbruntime‚20915  _runtime‚20915’"JTÉÆ0à _step‚957 & lr-AdamW/pg1‚9.343758671342614e-06 & lr-AdamW/pg2‚9.343758671342614e-06 trainer/global_step‚11499 % _timestamp‚1.7848601522800565e+09 _runtime‚20915.637307269 ²hÎÇ0È & lr-AdamW/pg1‚9.343758671342614e-06 & lr-AdamW/pg2‚9.343758671342614e-06 trainer/global_step‚11499 % _timestamp‚1.7848601522800565e+09 _runtime‚20915.637307269 _step‚957½•e(ù4È0/ _wandbruntime‚20915  _runtime‚20915—x½Á'É0P UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106815379 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3312532670556160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99891620 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11984643617014 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3312532644382720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206707005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3311226213343984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30647275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35063969 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚905943383611392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11499 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12003397649408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3324536042032128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3312532644382720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35063665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65711249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚920623158417408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99891626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3323210882388966 _step‚958 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106805315 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3324536042032128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚920623567289856 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3311226238771952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12004743085568 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak 8OJT‚660 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ºÍ(Ê0”P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚920623567289856 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3311226238771952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12004743085568 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106815379 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3312532670556160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99891620 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11984643617014 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3312532644382720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206707005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3311226213343984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30647275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35063969 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚905943383611392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11499 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12003397649408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3324536042032128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3312532644382720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35063665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65711249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚920623158417408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99891626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3323210882388966 _step‚958 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106805315 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3324536042032128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚959 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3312811579846656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106814397 trainer/global_step‚11499 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3312811579846656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3324817345447424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99899987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12005765600768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14681095037440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3324815973830656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65716582 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3323489334925128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3312811553673216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206714384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206724468 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 % _timestamp‚1.7848601537577524e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚920703024240640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚20917.115420547 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚906021969977344 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚906022338068480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11985664418766 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206714384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206724468 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 % _timestamp‚1.7848601537577524e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚920703024240640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚20917.115420547 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚906021969977344 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚906022338068480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11985664418766 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚959 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3312811579846656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106814397 trainer/global_step‚11499 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3312811579846656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3324817345447424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99899987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12005765600768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14681095037440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3324815973830656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65716582 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3323489334925128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3312811553673216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181¿m&÷"4Î0/ _wandbruntime‚20917  _runtime‚20917¯¡tYÏ0‰ 0loss/FBt1/detection_loss‚0.35906892396310613 _step‚960 /loss/FRt1/assignment_loss‚1.686801795522834 3loss/SRqqt2/assignment_loss‚0.37101434050322785 1loss/SRqqt1/detection_loss‚0.8992241688348446 /loss/FBt2/detection_loss‚0.4101358500076458 epoch‚41 2loss/SRqqt1/assignment_loss‚0.5028860343987804 2loss/FBt2/assignment_loss‚0.008262083369376082 1loss/SRqqt2/detection_loss‚0.6107098290834653 %loss/total_loss‚7.379857400443941 % _timestamp‚1.7848601537933753e+09 _runtime‚20917.150591563 0loss/FRt2/assignment_loss‚0.6290881963921529 4loss/FBt1/assignment_loss‚1.3780445878945748e-05 /loss/FRt1/detection_loss‚0.5182551063454172 trainer/global_step‚11499 /loss/FRt2/detection_loss‚1.3843972915772111>hrbÐ0Ž 1loss/SRqqt2/detection_loss‚0.6107098290834653 %loss/total_loss‚7.379857400443941 % _timestamp‚1.7848601537933753e+09 _runtime‚20917.150591563 0loss/FRt2/assignment_loss‚0.6290881963921529 4loss/FBt1/assignment_loss‚1.3780445878945748e-05 /loss/FRt1/detection_loss‚0.5182551063454172 trainer/global_step‚11499 /loss/FRt2/detection_loss‚1.3843972915772111 0loss/FBt1/detection_loss‚0.35906892396310613 _step‚960 /loss/FRt1/assignment_loss‚1.686801795522834 3loss/SRqqt2/assignment_loss‚0.37101434050322785 1loss/SRqqt1/detection_loss‚0.8992241688348446 /loss/FBt2/detection_loss‚0.4101358500076458 epoch‚41 2loss/SRqqt1/assignment_loss‚0.5028860343987804 2loss/FBt2/assignment_loss‚0.008262083369376082ÀM_HbÑ0‚(:’ œ‹Ó»»³ gpu.0.gpu‚96.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚266.618' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.87266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ý—QáÒ0‚(:Ä  œ‹Ó‹±ß³ disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚104.8154296875 disk.drbd1289.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.drbd181…4|Jn7.out‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.sda2.in‚ 0.03515625 network.recv‚ 1.502362e+06 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0( proc.memory.availableMB‚ 1933.7578125* proc.memory.percent‚11.972570419311523 proc.cpu.threads‚17 network.sent‚ 2.0133372e+07 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.sda.out‚ 850.04296875# disk.dm-0.out‚53070.93310546875 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625% disk./.usageGB‚1260.8666458129883 disk.drbd2078.in‚0 disk.drbd1673.in‚0$ memory_percent‚92.13151931762695 cpu‚25.33942429303149( disk./.usagePercent‚70.5142634096316 disk.drbd1938.out‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚53070.93310546875 disk.drbd2118.in‚0 disk.sda2.out‚ 850.04296875òÔ ›Ó0‚(: œ‹Ó–€¸³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.689' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.89633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï^_$@Ô0‚(:µ  œ‹Ó„âÙ³ disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.sda.out‚850.5 disk.drbd1828.out‚0* proc.memory.percent‚11.972570419311523 disk.drbd1510.out‚0 disk.drbd1965.out‚0 network.sent‚ 2.0195437e+07 disk.sda1.out‚0 network.recv‚ 1.504106e+06) disk./.usagePercent‚70.51426234294239 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2078.out‚0 disk.drbd1356.in‚0$ disk./.usageGB‚1260.866626739502 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 119.03125 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚850.5 disk.drbd1938.in‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 1933.67578125 disk.drbd1285.in‚0# disk.dm-0.out‚53071.68310546875 disk.drbd2078.in‚0 disk.drbd1356.out‚0 cpu‚25.339396207887617 proc.cpu.threads‚17# memory_percent‚92.1318531036377& disk.nvme0n1.out‚53071.68310546875 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.rbd1.out‚ 104.84765625 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 119.03125‡2.‘žÕ0‚(:“ Ÿœ‹Ó¹¶É³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚305.206' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.73533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´u7KÖ0‚(:À  Ÿœ‹Óïþæ³ disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 cpu‚25.33932015114192 network.recv‚ 1.504172e+06 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0# memory_percent‚92.1318531036377 disk.rbd1.out‚ 104.94921875 disk.drbd1552.in‚0 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.972570419311523 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 850.9921875 disk.drbd1931.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚53072.05029296875 disk.sda1.in‚0 network.sent‚ 2.0195503e+07$ disk./.usageGB‚1260.866626739502 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.sda.out‚ 850.9921875# disk.dm-0.out‚53072.05029296875) proc.memory.availableMB‚ 1933.67578125 disk.nvme0n1.in‚ 119.03125) disk./.usagePercent‚70.51426234294239 disk.drbd1965.out‚0 disk.rbd1.in‚0ÿXYÏ×0‚(:’ ®œ‹ÓœþƳ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚291.404' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.13466666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mð¾OØ0‚(:Ä  ®œ‹Óíÿè³ disk.drbd2111.in‚0 disk.drbd1938.in‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0$ memory_percent‚92.13188489278158 disk.drbd1285.out‚0 cpu‚25.339315832629232 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.sda.out‚ 851.32421875# disk.dm-0.out‚53072.98974609375) proc.memory.availableMB‚ 1933.66796875 network.sent‚ 2.0198527e+07 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 851.32421875 disk.dm-0.in‚ 119.03125 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.out‚0 disk.drbd1931.out‚0% disk./.usageGB‚1260.8649597167969& disk.nvme0n1.out‚53072.98974609375 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 network.recv‚ 1.50512e+06) disk./.usagePercent‚70.51416911430593 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.972570419311523 disk.rbd1.out‚ 105.19140625 disk.drbd1226.in‚0 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd1550.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd1510.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1289.out‚0ÀÜàÙ0‚(:’ ½œ‹Óˆ‘ª³ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.283' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.09433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º!î†MÚ0‚(:  ½œ‹Óš‚̳ disk.drbd2118.in‚0 network.recv‚ 1.505186e+06 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625$ memory_percent‚92.13188489278158 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0% disk./.usageGB‚1260.8649597167969 disk.drbd1550.in‚0 disk.drbd1673.out‚0# disk.dm-0.out‚53073.23583984375 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1933.66796875 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.out‚0 disk.sda.out‚ 851.7109375 disk.sda2.out‚ 851.7109375 disk.drbd1931.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd2078.out‚0) disk./.usagePercent‚70.51416911430593 disk.dm-0.in‚ 119.03125* proc.memory.percent‚11.972570419311523 disk.rbd1.out‚ 105.19140625 disk.sda1.in‚0 cpu‚25.33924076709858 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 network.sent‚ 2.0198593e+07 disk.drbd1584.in‚0 disk.drbd1817.out‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd2041.out‚0& disk.nvme0n1.out‚53073.23583984375 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0·âIÌPÛ0‚(:Å  Ìœ‹ÓÛÝÖ³ disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2942.37890625 disk.sda1.in‚0 disk.sda2.out‚ 852.01171875 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 network.recv‚ 1.506244e+06% disk./.usageGB‚1260.8649597167969 disk.rbd1.in‚0 disk.rbd1.out‚ 105.43359375 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 proc.cpu.threads‚17 network.sent‚ 2.0201632e+07 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 852.01171875 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 119.03125) disk./.usagePercent‚70.51416911430593 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 1933.66796875 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚53073.90966796875$ memory_percent‚92.13188489278158* proc.memory.percent‚11.972570419311523 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.nvme0n1.in‚ 119.03125& disk.nvme0n1.out‚53073.90966796875 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 cpu‚25.339188795394847 disk.drbd1550.out‚0 disk.drbd1289.in‚0}5'jÜ0‚(:’ Ìœ‹Ó¡‹³³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.369' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.78966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vƒÉì4Ý0/ _wandbruntime‚21002  _runtime‚21002ÖžYÈÞ0 & lr-AdamW/pg1‚8.983658274823272e-06 & lr-AdamW/pg2‚8.983658274823272e-06 trainer/global_step‚11549 % _timestamp‚1.7848602387697022e+09 _runtime‚21002.12788745 _step‚961¯xÌiÍß0Ç _runtime‚21002.12788745 _step‚961 & lr-AdamW/pg1‚8.983658274823272e-06 & lr-AdamW/pg2‚8.983658274823272e-06 trainer/global_step‚11549 % _timestamp‚1.7848602387697022e+09Áû3°a4à0/ _wandbruntime‚21002  _runtime‚21002…C(á0“P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107271331 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30779275 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924615736078336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3337202074960574 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12034402888536 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3326478108907520 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3326478135080960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100309970 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3337200704921320 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3325166302032784 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12055932641792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3325166327460752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14742612713984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35212172 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14742666078208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35212515 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 _runtime‚21002.127965275 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12035747499822 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12054587205632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3338532696113152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207581307 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100309976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65991795 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12055932641792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30779280 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107261267 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3338534067722752fÆ2(â0˜P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚343 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚909873070000128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 trainer/global_step‚11549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107261267 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3338534067722752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107271331 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30779275 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924615736078336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3337202074960574 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12034402888536 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3326478108907520 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3326478135080960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100309970 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3337200704921320 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3325166302032784 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12055932641792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3325166327460752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14742612713984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35212172 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14742666078208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35212515 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 _runtime‚21002.127965275 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚348 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12035747499822 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12054587205632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3338532696113152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207581307 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100309976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65991795 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12055932641792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30779280 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3337481897536102 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12055609713664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207598770 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65997373 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚21003.595627506 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12056955156992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14743919468544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3337480527496736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207588686 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107280427 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14743861917184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 trainer/global_step‚11549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚909951288274944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30781915 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _step‚963 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3325445129234528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100318337 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3338813999528448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100318337 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65997020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30781920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207598770 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12056955156992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207588686 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12055609713664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.784860240238035e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚21003.595627506 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12056955156992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14743919468544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3337480527496736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207588686 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107280427 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14743861917184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 trainer/global_step‚11549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚909951288274944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30781915 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _step‚963 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3325445129234528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100318337 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3338813999528448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100318337 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65997020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30781920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207598770 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12056955156992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207588686 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12055609713664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.784860240238035e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3337481897536102 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12055609713664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207598770 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65997373ÃZ£xX4æ0/ _wandbruntime‚21003  _runtime‚21003(÷­Lç0‰ 0loss/FRt1/assignment_loss‚1.5758124788403511 4loss/FBt1/assignment_loss‚0.00024194282994680888 2loss/FBt2/assignment_loss‚0.013442467384704757 1loss/SRqqt1/detection_loss‚0.7891042399314756 /loss/FRt1/detection_loss‚0.5158393602604083 1loss/SRqqt2/detection_loss‚0.5765283554318799 epoch‚41 trainer/global_step‚11549 _runtime‚21003.638444481 2loss/SRqqt1/assignment_loss‚0.5234898946531175 /loss/FBt1/detection_loss‚0.7094794518369085 /loss/FBt2/detection_loss‚0.4251802695214576 %loss/total_loss‚7.575373279402776 % _timestamp‚1.7848602402812579e+09 0loss/FRt2/assignment_loss‚0.5824099175222628 3loss/SRqqt2/assignment_loss‚0.46953646592528886 /loss/FRt2/detection_loss‚1.3943084352649748 _step‚964™ü2è0Ž 0loss/FRt1/assignment_loss‚1.5758124788403511 4loss/FBt1/assignment_loss‚0.00024194282994680888 2loss/FBt2/assignment_loss‚0.013442467384704757 1loss/SRqqt1/detection_loss‚0.7891042399314756 /loss/FRt1/detection_loss‚0.5158393602604083 1loss/SRqqt2/detection_loss‚0.5765283554318799 epoch‚41 trainer/global_step‚11549 _runtime‚21003.638444481 2loss/SRqqt1/assignment_loss‚0.5234898946531175 /loss/FBt1/detection_loss‚0.7094794518369085 /loss/FBt2/detection_loss‚0.4251802695214576 %loss/total_loss‚7.575373279402776 % _timestamp‚1.7848602402812579e+09 0loss/FRt2/assignment_loss‚0.5824099175222628 3loss/SRqqt2/assignment_loss‚0.46953646592528886 /loss/FRt2/detection_loss‚1.3943084352649748 _step‚964Ä ½žé0‚(:“ Ûœ‹Ó±ÙÁ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.051' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.68366666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00èå`Hê0‚(:½  Ûœ‹ÓÔ•å³$ memory_percent‚92.13205973307291# proc.memory.rssMB‚ 2942.37890625 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 cpu‚25.339136834399348 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd2041.in‚0$ proc.memory.availableMB‚1933.625 disk.rbd1.out‚ 105.53515625% disk./.usageGB‚1260.8688430786133 disk.drbd1673.out‚0# disk.dm-0.out‚53074.12060546875 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 network.sent‚ 2.0201698e+07 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚53074.12060546875* proc.memory.percent‚11.972570419311523 disk.sda2.out‚ 852.4140625 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 119.03125 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.sda.out‚ 852.4140625 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51438629222793 disk.drbd1938.out‚0 disk.drbd2111.out‚0 network.recv‚ 1.50631e+06 disk.drbd1289.in‚0 disk.drbd1938.in‚0¶0¹*ë0‚(:’ ꜋Ó¾ðų gpu.0.gpu‚98.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚253.275' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.42500000000001 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*¢#wHì0‚(:½  ꜋Ó³Ùà³ disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2942.37890625 network.recv‚ 1.507469e+06 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.sda1.out‚0 disk.sda.out‚ 852.6640625 disk.sda2.out‚ 852.6640625 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 119.03125% disk./.usageGB‚1260.8688430786133 disk.drbd1289.out‚0 disk.drbd1965.in‚0 network.sent‚ 2.0207368e+07 disk.drbd1552.out‚0! disk.dm-0.out‚53074.486328125 disk.drbd1510.in‚0 disk.sda.in‚ 0.03515625$ disk.nvme0n1.out‚53074.486328125 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 105.59765625 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0' proc.memory.availableMB‚ 1933.578125 proc.cpu.threads‚17) disk./.usagePercent‚70.51438629222793 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.dm-0.in‚ 119.03125* proc.memory.percent‚11.972570419311523 disk.drbd1226.in‚0 cpu‚25.339097181121865 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625$ memory_percent‚92.13225046793619²…Mží0‚(:“ ùœ‹Ó®°ä³ gpu.0.gpu‚36.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚314.123' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.70766666666667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÁWJî0‚(:¿  ùœ‹ÓÚ©‚´ disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1226.in‚0 network.recv‚ 1.507535e+06 disk.sda.in‚ 0.03515625 disk.drbd1226.out‚0 disk.sda.out‚ 853.1171875 disk.drbd1289.out‚0 disk.rbd1.out‚ 105.59765625 disk.drbd1510.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2942.37890625$ disk.nvme0n1.out‚53074.677734375 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51438735891713 disk.nvme0n1.in‚ 119.03125 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0$ memory_percent‚92.13124910990396 disk.dm-0.in‚ 119.03125 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0% disk./.usageGB‚1260.8688621520996 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 cpu‚25.339044986636264 proc.cpu.threads‚17! disk.dm-0.out‚53074.677734375 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 853.1171875 network.sent‚ 2.0207434e+07* proc.memory.percent‚11.972570419311523 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1933.82421875ÁÄqï0‚(:’ ˆ‹Ó÷Ƨ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚295.877' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.62566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ½lðQð0‚(:Æ  ˆ‹Ó€Ëų disk.nvme0n1.in‚ 119.03125 disk.drbd1510.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1933.82421875 disk.drbd2118.in‚0* proc.memory.percent‚11.972570419311523( disk./.usagePercent‚70.5143867189036 disk.drbd1584.in‚0 disk.drbd1356.out‚0 proc.cpu.threads‚17 network.recv‚ 1.509457e+06 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 network.sent‚ 2.0270124e+07 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 853.40234375 disk.drbd2078.in‚0 cpu‚25.338982861785393 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0# disk.dm-0.out‚53075.20068359375# proc.memory.rssMB‚ 2942.37890625% disk./.usageGB‚1260.8688507080078 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚53075.20068359375 disk.drbd2118.out‚0 disk.sda.out‚ 853.40234375 disk.drbd1938.in‚0 disk.rbd1.out‚105.6298828125$ memory_percent‚92.13124910990396{Üññ0‚(:’ —‹Óàáß³ gpu.0.gpu‚91.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚246.478' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.15933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>PûòNò0‚(:à  —‹Óâ”þ³ disk.drbd1510.in‚0 disk.sda2.out‚ 853.74609375* proc.memory.percent‚11.972570419311523 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 853.74609375 disk.drbd1828.in‚0 disk.rbd1.out‚105.8173828125 disk.dm-0.in‚ 119.03125 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0# disk.dm-0.out‚53075.74755859375 disk.rbd1.in‚0' proc.memory.availableMB‚ 1934.078125 cpu‚25.338942194660568 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.sda1.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0( disk./.usagePercent‚70.5143867189036 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 119.03125 network.recv‚ 1.509523e+06 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2942.37890625% disk./.usageGB‚1260.8688507080078 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0& disk.nvme0n1.out‚53075.74755859375 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 network.sent‚ 2.027019e+07 disk.drbd1673.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0$ memory_percent‚92.13021596272787ém¡4ó0/ _wandbruntime‚21088  _runtime‚21088‚êîÈô0 & lr-AdamW/pg2‚8.630056792210688e-06 trainer/global_step‚11599 $ _timestamp‚1.784860324998349e+09 _runtime‚21088.355588873 _step‚965 & lr-AdamW/pg1‚8.630056792210688e-069”‡Íõ0Ç & lr-AdamW/pg1‚8.630056792210688e-06 & lr-AdamW/pg2‚8.630056792210688e-06 trainer/global_step‚11599 $ _timestamp‚1.784860324998349e+09 _runtime‚21088.355588873 _step‚965Å´\(õ4ö0/ _wandbruntime‚21088  _runtime‚21088pTTi÷0’P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3340423573432320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100728326 O@DeviceStatsMonitor.on_tra{<6B¯&in_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12107122198016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107717219 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12107122198016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107727283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30911275 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3352529350194176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚913802388297728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3351191897492928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3340423573432320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66270285 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107717219 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3339106416149552 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3340423599605760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14805117300736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 % _timestamp‚1.7848603249994314e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208455609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12105776761856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚966 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107727283 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100728326 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66270659 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100728320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30911280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12105776761856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12086851382630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14805061839360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3339106390721584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35359379 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚369 _runtime‚21088.35696137 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208445539 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3352529350194176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928607505598464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3352530721803776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12085506771344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3340423599605760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100728320 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3352529350194176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚913802388297728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3351191897492928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3340423573432320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66270285 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107717219 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3339106416149552 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3340423599605760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14805117300736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 % _timestamp‚1.7848603249994314e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208455609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12105776761856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚966 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107727283 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100728326 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66270659 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100728320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30911280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12105776761856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12086851382630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14805061839360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3339106390721584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35359379 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚369 _runtime‚21088.35696137 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208445539 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3352529350194176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928607505598464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3352530721803776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12085506771344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3340423599605760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100728320 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3340423573432320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100728326 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12107122198016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107717219 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12107122198016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107727283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30911275 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Æ[¼œá4ù0/ _wandbruntime‚21089  _runtime‚21089+LTŽ#'ú0N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100736693 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3351471720068344 _runtime‚21089.832752459 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3340702482722816 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3340702508896256 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12108144713216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12106799269888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 trainer/global_step‚11599 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3352810653609472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3351473090107710 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12086527572984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30913915 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3340702508896256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12106799269888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66275861 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚913880974663680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚913880606572544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3339385217923328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208462988 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107736379 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚967 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208462988 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66276232 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12108144713216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100736687 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3339385192495360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚371 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107726301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208473072 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107726301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928687341831168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14806367167488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100736687 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚366 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3340702482722816 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3340702508896256 ]JDeviceStatí~b¤nsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12108144713216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12106799269888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 trainer/global_step‚11599 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3352810653609472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3351473090107710 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12086527572984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30913915 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3340702508896256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12106799269888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66275861 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚913880974663680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚913880606572544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3339385217923328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208462988 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107736379 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚967 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208462988 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66276232 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12108144713216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100736687 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3339385192495360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚371 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107726301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208473072 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107726301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928687341831168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825Ç»µ4ü0/ _wandbruntime‚21089  _runtime‚21089?‰øý0‰ 0loss/FBt2/detection_loss‚0.32638378968416637 3loss/SRqqt2/assignment_loss‚0.48093481370411395 3loss/FBt1/assignment_loss‚0.0007729118495351995 2loss/FBt2/assignment_loss‚0.012782260073569451 1loss/SRqqt2/detection_loss‚0.5687940813536726 /loss/FBt1/detection_loss‚0.5804313327073408 0loss/FRt2/assignment_loss‚0.6494043063858281 /loss/FRt1/detection_loss‚0.4879227609628969 epoch‚41 _runtime‚21089.869201128 _step‚968 0loss/FRt1/assignment_loss‚1.6039769048116121 2loss/SRqqt1/assignment_loss‚0.4731865054699535 /loss/FRt2/detection_loss‚1.3612854768491716 % _timestamp‚1.7848603265119853e+09 1loss/SRqqt1/detection_loss‚0.9391398231232335 %loss/total_loss‚7.485014966975093 trainer/global_step‚11599š³¢¼þ0Ž /loss/FBt1/detection_loss‚0.5804313327073408 0loss/FRt2/assignment_loss‚0.6494043063858281 /loss/FRt1/detection_loss‚0.4879227609628969 epoch‚41 _runtime‚21089.869201128 _step‚968 0loss/FRt1/assignment_loss‚1.6039769048116121 2loss/SRqqt1/assignment_loss‚0.4731865054699535 /loss/FRt2/detection_loss‚1.3612854768491716 % _timestamp‚1.7848603265119853e+09 1loss/SRqqt1/detection_loss‚0.9391398231232335 %loss/total_loss‚7.485014966975093 trainer/global_step‚11599 0loss/FBt2/detection_loss‚0.32638378968416637 3loss/SRqqt2/assignment_loss‚0.48093481370411395 3loss/FBt1/assignment_loss‚0.0007729118495351995 2loss/FBt2/assignment_loss‚0.012782260073569451 1loss/SRqqt2/detection_loss‚0.5687940813536726È}”³Iÿ0‚(:¾  ¦‹Óáêü³ disk.drbd1510.in‚0 disk.drbd1550.out‚0 cpu‚25.338879065939654 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚105.8798828125 network.sent‚ 2.0275789e+07 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.out‚ 854.109375 disk.sda2.out‚ 854.109375 disk.drbd1552.out‚0$ memory_percent‚92.13153521219888 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2942.37890625 network.recv‚ 1.510679e+06 disk.drbd1965.in‚0 disk.drbd1817.out‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.972570419311523( disk./.usagePercent‚70.5143856522144 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.03515625$ disk.nvme0n1.out‚53076.138671875) proc.memory.availableMB‚ 1933.75390625! disk.dm-0.out‚53076.138671875% disk./.usageGB‚1260.8688316345215 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0£-€€1‚(:’ ¦‹ÓÔÇÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚109.526' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.50866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üðpI’1‚(:‡ µ‹ÓÄüÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚277.551' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.517 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î\´ÏK‚1‚(:À  µ‹Óú™ü³$ memory_percent‚92.13148752848306 disk.dm-0.in‚ 119.03125 disk.drbd1584.in‚0 disk.drbd1828.in‚0( disk./.usagePercent‚70.5143856522144 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0' proc.memory.availableMB‚ 1933.765625 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8688316345215 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 cpu‚25.338863311054766 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd1285.in‚0# proc.memory.rssMB‚ 2942.37890625* proc.memory.percent‚11.972570419311523 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1931.out‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1289.in‚0 disk.drbd1817.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 network.recv‚ 1.511633e+06 disk.sda.out‚ 854.43359375 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 854.43359375 disk.rbd1.out‚105.8798828125 network.sent‚ 2.0281467e+07 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0! disk.dm-0.out‚53076.384765625$ disk.nvme0n1.out‚53076.384765625$¶ìœƒ1‚(:‘ Ä‹Óå£Û³ gpu.0.gpu‚12.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚308.23' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.74333333333334 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µe¾uF„1‚(:»  Ä‹Óñ³÷³ disk.drbd1226.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2942.37890625* proc.memory.percent‚11.972570419311523 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda2.in‚ 0.03515625" disk.dm-0.out‚53076.7861328125( disk./.usagePercent‚70.5143856522144 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 119.03125% disk.nvme0n1.out‚53076.7861328125$ memory_percent‚92.13150342305502% disk./.usageGB‚1260.8688316345215 disk.sda.out‚ 854.71875 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 cpu‚25.33881221437268 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 105.912109375) proc.memory.availableMB‚ 1933.76171875 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 network.sent‚ 2.0341489e+07 disk.drbd1552.out‚0 disk.sda2.out‚ 854.71875 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 119.03125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 network.recv‚ 1.51361e+06 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2041.in‚0fš‚…1‚(:’ Ó‹ÓÎʳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.975' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.99166666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä?öçN†1‚(:à  Ó‹ÓŸ“ç³ disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0$ memory_percent‚92.13150342305502 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0% disk.nvme0n1.out‚53077.1337890625 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.972570419311523 proc.cpu.threads‚17 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 network.sent‚ 2.0341555e+07 network.recv‚ 1.513676e+06 disk.sda2.out‚ 855.17578125" disk.dm-0.out‚53077.1337890625 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚ 106.013671875 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0% disk./.usageGB‚1260.8688316345215 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1933.76171875( disk./.usagePercent‚70.5143856522144 disk.sda.out‚ 855.17578125 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 cpu‚25.338749596136505 disk.drbd1584.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0b¢0A‡1‚(:’ â‹ÓÚüê³ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚271.082' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.36066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V‰Dˆ1‚(:¹  â‹ÓÒ¹Š´ disk.drbd2078.out‚0 disk.drbd1356.in‚0 network.recv‚ 1.514635e+06 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1356.out‚0' proc.memory.availableMB‚ 1933.734375# proc.memory.rssMB‚ 2942.37890625 disk.drbd1584.out‚0% disk./.usageGB‚1260.8688316345215 disk.drbd2078.in‚0# memory_percent‚92.1316146850586 disk.drbd1673.out‚0 disk.drbd1828.out‚0$ disk.nvme0n1.out‚53077.447265625 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 network.sent‚ 2.0344584e+07 disk.drbd1289.in‚0 disk.dm-0.in‚ 119.03125( disk./.usagePercent‚70.5143856522144 disk.drbd2111.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 106.076171875 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 cpu‚25.33870966802125 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.972570419311523 disk.drbd2041.out‚0 disk.sda.out‚ 855.578125 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.out‚0 disk.drbd1510.out‚0! disk.dm-0.out‚53077.447265625 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.sda2.out‚ 855.578125 disk.drbd1931.out‚0 disk.drbd1938.out‚0Zâã…H‰1‚(:½  ñ‹ÓÐû׳ disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 855.90234375 disk.drbd1289.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2942.37890625 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1931.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.972570419311523 disk.drbd2111.in‚0 network.sent‚ 2.0344734e+07 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125' proc.memory.availableMB‚ 1933.734375 disk.drbd2111.out‚0# memory_percent‚92.1316146850586 disk.drbd1938.in‚0( disk./.usagePercent‚70.5143856522144 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 855.90234375! disk.dm-0.out‚53077.634765625 disk.drbd2078.in‚0 disk.drbd1356.in‚0 cpu‚25.33865967920577 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 network.recv‚ 1.514785e+06 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚ 106.076171875 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8688316345215$ disk.nvme0n1.out‚53077.634765625 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625lªm2žŠ1‚(:“ ñ‹Óˆ„²³ gpu.0.gpu‚93.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.235' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.07833333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦„c¼4‹1/ _wandbruntime‚21175  _runtime‚21175¼ë<ÉÉŒ1à & lr-AdamW/pg1‚8.283000572557804e-06 & lr-AdamW/pg2‚8.283000572557804e-06 trainer/global_step‚11649 % _timestamp‚1.7848604117116628e+09 _runtime‚21175.068897246 _step‚969n–Î1È & lr-AdamW/pg2‚8.283000572557804e-06 trainer/global_step‚11649 % _timestamp‚1.7848604117116628e+09 _runtime‚21175.068897246 _step‚969 & lr-AdamW/pg1‚8.283000572557804e-06ÉûµDè4Ž1/ _wandbruntime‚21175  _runtime‚21175™2„'(1“P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108173171 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108173171 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35506683 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚917731338504192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3365184460103790 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108183235 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101146676 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101146670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66550323 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3353046504838352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚365 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209329911 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚21175.070077654 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3366527375884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108183235 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3353046479410384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101146670 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3366527375884800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12156966318080 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3354369037957120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14867649691648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31043275 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3366526004275200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070j{Ï¡21˜P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108173171 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108173171 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35506683 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚917731338504192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3365184460103790 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108183235 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101146676 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101146670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66550323 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3353046504838352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚365 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209329911 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚21175.070077654 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3366527375884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108183235 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3353046479410384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101146670 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3366527375884800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12156966318080 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3354369037957120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14867649691648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31043275 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 _step‚970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3366526004275200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070ʾ4Îã4‘1/ _wandbruntime‚21176  _runtime‚21176ˆ;®T#'’1N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66555882 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3354647973421056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35509604 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚363 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚917810292961280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12137631455792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚932678776827392 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209347374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108182253 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14868903217152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3353325306612128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3353325281184160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _step‚971 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3354647947247616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101155043 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3366805936073728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108192331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3366807307690496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108182253 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101155037 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31045920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3365462912639952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚932679196178432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3365464282679318 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14868851957248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66555519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚917809924870144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108192331 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12138976067190 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 trainer/global_step‚11649 _runtime‚21176.547905534 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12159334269440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12157988826112 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209347374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108182253 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14868903217152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3353325306612128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3353325281184160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _step‚971 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3354647947247616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101155043 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3366805936073728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108192331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3366807307690496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108182253 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101155037 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31045920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3365462912639952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚932679196178432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3365464282679318 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14868851957248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66555519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚917809924870144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108192331 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12138976067190 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 trainer/global_step‚11649 _runtime‚21176.547905534 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12159334269440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12157988826112 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚363 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653Ëãé·4”1/ _wandbruntime‚21176  _runtime‚21176·qI•1‚ %loss/total_loss‚7.699166447828729 epoch‚41 1loss/SRqqt1/assignment_loss‚0.375166067760022 /loss/FBt2/assignment_loss‚0.006827629343856 /loss/FRt2/detection_loss‚1.4513483731985322 1loss/SRqqt1/detection_loss‚0.9768630919686985 /loss/FBt1/detection_loss‚0.5198713831312489 0loss/FBt2/detection_loss‚0.49737931257748924 trainer/global_step‚11649 _runtime‚21176.583815306 2loss/FBt1/assignment_loss‚0.041768353207014816 1loss/SRqqt2/detection_loss‚0.5114784929894254 _step‚972 0loss/FRt1/assignment_loss‚1.6161133555861025 /loss/FRt2/assignment_loss‚0.651757179841293 3loss/SRqqt2/assignment_loss‚0.49488102761004454 /loss/FRt1/detection_loss‚0.5557121806150008 $ _timestamp‚1.784860413226567e+09Z| –1‡ 2loss/FBt1/assignment_loss‚0.041768353207014816 1loss/SRqqt2/detection_loss‚0.5114784929894254 _step‚972 0loss/FRt1/assignment_loss‚1.6161133555861025 /loss/FRt2/assignment_loss‚0.651757179841293 3loss/SRqqt2/assignment_loss‚0.49488102761004454 /loss/FRt1/detection_loss‚0.5557121806150008 $ _timestamp‚1.784860413226567e+09 %loss/total_loss‚7.699166447828729 epoch‚41 1loss/SRqqt1/assignment_loss‚0.375166067760022 /loss/FBt2/assignment_loss‚0.006827629343856 /loss/FRt2/detection_loss‚1.4513483731985322 1loss/SRqqt1/detection_loss‚0.9768630919686985 /loss/FBt1/detection_loss‚0.5198713831312489 0loss/FBt2/detection_loss‚0.49737931257748924 trainer/global_step‚11649 _runtime‚21176.583815306Ìð;—1‚(:’ €ž‹Ó۱Ⳡgpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.312' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.77066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÜ|p˜1‚(:Á  €ž‹Ó”ž´% proc.memory.availableMB‚ 1933.6875 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 network.recv‚ 1.515633e+06 disk.nvme0n1.in‚ 119.03125 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚106.1083984375 disk.drbd1931.in‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.972570419311523 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0# disk.dm-0.out‚53077.98583984375 disk.drbd1965.in‚0$ memory_percent‚92.13180541992188( disk./.usagePercent‚70.5143856522144 disk.drbd1584.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚53077.98583984375 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0# pƒË…®Üroc.memory.rssMB‚ 2942.37890625 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 cpu‚25.33860797237943 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 856.25390625 disk.drbd1552.out‚0 disk.drbd2041.out‚0 network.sent‚ 2.0350292e+07 disk.drbd1226.in‚0 disk.sda2.out‚ 856.25390625 disk.drbd1938.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2118.out‚0% disk./.usageGB‚1260.8688316345215 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0]òXtž™1‚(:“ ž‹Ó¾¶³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚308.179' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.72633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€"¹ÍMš1‚(:  ž‹ÓÜÁس disk.drbd1673.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0# disk.dm-0.out‚53078.31005859375 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 network.recv‚ 1.516743e+06 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0( proc.memory.availableMB‚ 1933.6328125 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1510.out‚0 cpu‚25.338557930676053* proc.memory.percent‚11.972570419311523 disk.sda.out‚ 856.578125 disk.rbd1.out‚106.2099609375 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2942.37890625 network.sent‚ 2.0355992e+07) disk./.usagePercent‚70.51433573115963 disk.drbd1673.out‚0 disk.sda2.out‚ 856.578125 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd2078.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2078.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.13202794392905 proc.cpu.threads‚17% disk./.usageGB‚1260.8679389953613 disk.drbd2118.in‚0 disk.drbd1965.in‚0& disk.nvme0n1.out‚53078.31005859375 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0þNê`ž›1‚(:“ žž‹ÓÔ¦Ú³ gpu.0.gpu‚66.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.925' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.30833333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Ó~½Mœ1‚(:  žž‹Ó¹¶õ³ disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 cpu‚25.33850659950471 disk.drbd2041.in‚0) proc.memory.availableMB‚ 1933.62890625) disk./.usagePercent‚70.51433701118668 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚106.2724609375 disk.drbd1673.in‚0 disk.drbd1510.in‚0$ memory_percent‚92.13204383850098 network.sent‚ 2.041603e+07$ disk./.usageGB‚1260.867961883545 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.in‚0* proc.memory.percent‚11.972570419311523 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 119.03125 disk.sda1.out‚0 network.recv‚ 1.518562e+06 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.sda.out‚ 856.89453125 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0" disk.dm-0.out‚53078.5908203125 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚53078.5908203125# proc.memory.rssMB‚ 2942.37890625 proc.cpu.threads‚17 disk.sda2.out‚ 856.89453125 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0öZ>1‚(:’ ­ž‹ÓŒ²Ì³ gpu.0.gpu‚61.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚108.421' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.14033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûö.Kž1‚(:À  ­ž‹Ó¤ší³% disk./.usageGB‚1260.8678283691406 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.sda2.out‚ 857.43359375 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.rbd1.out‚106.2724609375 disk.drbd1285.in‚0% disk.nvme0n1.out‚53078.8173828125 disk.drbd1550.out‚0 cpu‚25.33845606687477 network.sent‚ 2.0416096e+07 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1965.in‚0% proc.memory.availableMB‚ 1933.4375 disk.sda2.in‚ 0.03515625 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2942.37890625* proc.memory.percent‚11.972570419311523 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0" disk.dm-0.out‚53078.8173828125 disk.sda1.in‚0 disk.sda.out‚ 857.43359375 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 network.recv‚ 1.518628e+06 disk.dm-0.in‚ 119.03125 disk.drbd2111.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0$ memory_percent‚92.13282267252603 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51432954436225 disk.nvme0n1.in‚ 119.03125âÎEb“Ÿ1‚(:ˆ ¼ž‹Óé—Ú³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.009' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j|Ü#I 1‚(:¾  ¼ž‹Óïðý³ disk.drbd1289.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1226.in‚0 disk.drbd1510.in‚0 cpu‚25.33840512130965$ disk./.usageGB‚1260.867847442627 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚53079.68603515625 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 106.3046875 disk.drbd2041.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.13282267252603 network.sent‚ 2.0419141e+07 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.nvme0n1.in‚ 119.03125# proc.memory.rssMB‚ 2942.37890625 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.sda2.out‚ 857.92578125 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.51433061105146 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 857.92578125 disk.drbd1828.out‚0 disk.drbd1285.in‚0 network.recv‚ 1.519642e+06# disk.dm-0.out‚53079.68603515625 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0% proc.memory.availableMB‚ 1933.4375 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1965.in‚0* proc.memory.percent‚11.972570419311523 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0„ù[~¡1‚(:’ Ëž‹Ó³Þá³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.773' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.59100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯u(±K¢1‚(:À  Ëž‹Óíу´* proc.memory.percent‚11.972570419311523 disk.drbd2111.out‚0 disk.drbd2118.out‚0 network.sent‚ 2.0424809e+07 disk.drbd1356.out‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚53080.35400390625# memory_percent‚92.1318213144938# disk.dm-0.out‚53080.35400390625 disk.drbd2041.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1965.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2942.37890625 cpu‚25.33835448004034 network.recv‚ 1.520625e+06 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 119.03125 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.sda2.out‚ 858.28515625 disk.drbd1938.in‚0 disk.rbd1.out‚ 106.40625 disk.drbd1817.in‚0) proc.memory.availableMB‚ 1933.68359375) disk./.usagePercent‚70.51443877333678 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda.out‚ 858.28515625% disk./.usageGB‚1260.8697814941406 disk.drbd1285.in‚0Â¥÷4£1/ _wandbruntime‚21260  _runtime‚21260Èœ¦É¤1à & lr-AdamW/pg2‚7.942535106983542e-06 trainer/global_step‚11699 % _timestamp‚1.7848604973701768e+09 _runtime‚21260.727432097 _step‚973 & lr-AdamW/pg1‚7.942535106983542e-06f_×pÎ¥1È _step‚973 & lr-AdamW/pg1‚7.942535106983542e-06 & lr-AdamW/pg2‚7.942535106983542e-06 trainer/global_step‚11699 % _timestamp‚1.7848604973701768e+09 _runtime‚21260.727432097ÍŸ”7£4¦1/ _wandbruntime‚21260  _runtime‚21260õ²ªÄ(§1“P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12208155874304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚921660656801792 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3366986593527152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108629123 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12189059148246 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 _runtime‚21260.731896932 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3380522658356224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3368314502481920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848604973711908e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14930152894976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3380522658356224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12208155874304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚921661024892928 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66829394 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚936591177787904 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3368314528655360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3368314528655360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108629123 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31175280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31175275 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101565020 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3379175652675398 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3366986568099184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚355 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35654469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101565020 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚936590762624000 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚11699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3380524029965824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210194143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224§!Ri(¨1˜P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3368314528655360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108629123 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31175280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31175275 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101565020 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3379175652675398 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3366986568099184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚355 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35654469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101565020 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚936590762624000 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚11699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3380524029965824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210194143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12208155874304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚921660656801792 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3366986593527152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108629123 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12189059148246 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47072768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 _runtime‚21260.731896932 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3380522658356224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3368314502481920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 % _timestamp‚1.7848604973711908e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14930152894976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3380522658356224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12208155874304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚921661024892928 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66829394 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚936591177787904 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3368314528655360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006Î)§”U4©1/ _wandbruntime‚21262  _runtime‚21262ÔPS泪1œN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31177920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3368593437945856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210211592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \HDeviceStatsMonitor.on_trai ]é§o$n_batch_end/inactive_split_bytes.all.allocated‚936671025647104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3380803961771520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚921739611258880 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3367265369872960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12188735338600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 trainer/global_step‚11699 _runtime‚21262.199215755 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108638205 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14931358934016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35657397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108648283 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 $ _timestamp‚1.784860498841629e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12190079949998 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210221676 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108638205 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108648283 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3368593411772416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12209178382336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66835317 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3368593411772416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12209178382336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66834961 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3379455475250926 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101573387 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚921739243167744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101573387 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210221676 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0_''«1¡N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66834961 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3379455475250926 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101573387 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚921739243167744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101573387 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210221676 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31177920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3368593437945856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210211592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚936671025647104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3380803961771520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚921739611258880 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3367265369872960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12188735338600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 trainer/global_step‚11699 _runtime‚21262.199215755 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108638205 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14931358934016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35657397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108648283 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 $ _timestamp‚1.784860498841629e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12190079949998 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210221676 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108638205 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108648283 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3368593411772416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12209178382336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66835317 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3368593411772416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12209178382336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3394519312437248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109085075 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3382259967006720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31307280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3394520684046848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67109107 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚925590343190528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3382259993180160 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3382259967006720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3393165475207752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109095139 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67108764 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3393166845247006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12238818419768 trainer/global_step‚11749 _runtime‚21347.594934441 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848605842343583e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚978 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31307275 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211078515 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109085075 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14992860276736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3394519312437248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109085075 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3382259967006720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31307280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3394520684046848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67109107 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚925590343190528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3382259993180160 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3382259967006720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3393165475207752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109095139 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67108764 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3393166845247006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12238818419768 trainer/global_step‚11749 _runtime‚21347.594934441 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7848605842343583e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚978 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31307275 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211078515 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109085075 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14992860276736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31309915 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101991737 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784860585709183e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11749 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3382538876297216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14994106337280 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3382538876297216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12261713381888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚940663089250816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12241183832806 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3393445297783168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3394800615852544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12261713381888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211085894 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3382538902470656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35804724 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109104235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31309920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211085894 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67114299 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12239839221408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12260367938560 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101991743 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109094157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3394800615852544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3394799244235776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚979 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101991743 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101991737 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14994159694336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211095978 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 I@DeviceStatsMonith œù¡"or.on_train_batch_end/segment.large_pool.current‚1181 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3393445297783168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3394800615852544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12261713381888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211085894 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3382538902470656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35804724 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109104235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31309920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211085894 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67114299 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12239839221408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12260367938560 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101991743 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109094157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3394800615852544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3394799244235776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚979 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101991743 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101991737 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14994159694336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211095978 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31309915 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101991737 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784860585709183e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11749 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3382538876297216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14994106337280ÓOh Õ4Ä1/ _wandbruntime‚21349  _runtime‚21349Ó¾< Å1† 2loss/SRqqt1/assignment_loss‚0.5034480994538634 /loss/FBt1/detection_loss‚0.6461365939202633 /loss/FBt2/detection_loss‚0.3502218149202079 % _timestamp‚1.7848605857480388e+09 0loss/FRt2/assignment_loss‚0.6435122401421142 1loss/FBt2/assignment_loss‚0.04009131187235022 1loss/SRqqt1/detection_loss‚0.8695511192585977 1loss/SRqqt2/detection_loss‚0.5650943694215975 epoch‚41 /loss/FRt1/detection_loss‚0.5165692350615263 _step‚980 0loss/FRt1/assignment_loss‚1.6359689271614957 2loss/SRqqt2/assignment_loss‚0.4766649056623213 3loss/FBt1/assignment_loss‚8.452572470046658e-06 /loss/FRt2/detection_loss‚1.3801229542354663 %loss/total_loss‚7.627390023682273 trainer/global_step‚11749 _runtime‚21349.1052221291S[Æ1‹ 0loss/FRt2/assignment_loss‚0.6435122401421142 1loss/FBt2/assignment_loss‚0.04009131187235022 1loss/SRqqt1/detection_loss‚0.8695511192585977 1loss/SRqqt2/detection_loss‚0.5650943694215975 epoch‚41 /loss/FRt1/detection_loss‚0.5165692350615263 _step‚980 0loss/FRt1/assignment_loss‚1.6359689271614957 2loss/SRqqt2/assignment_loss‚0.4766649056623213 3loss/FBt1/assignment_loss‚8.452572470046658e-06 /loss/FRt2/detection_loss‚1.3801229542354663 %loss/total_loss‚7.627390023682273 trainer/global_step‚11749 _runtime‚21349.105222129 2loss/SRqqt1/assignment_loss‚0.5034480994538634 /loss/FBt1/detection_loss‚0.6461365939202633 /loss/FBt2/detection_loss‚0.3502218149202079 % _timestamp‚1.7848605857480388e+09Ôj­ÃŽœÇ1‚(:‘ ´Ÿ‹Ó³ç¿³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.007' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.00233333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú_3¬OÈ1‚(:Ä  ´Ÿ‹ÓÒôç³ disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 861.44140625 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0# proc.memory.rssMB‚ 2942.37890625 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2078.in‚0 network.recv‚ 1.526803e+06 disk.drbd1285.in‚0" disk.dm-0.out‚53084.4267578125 disk.sda1.in‚0 cpu‚25.33802653245533 network.sent‚ 2.0502083e+07 disk.rbd1.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1552.out‚0 disk.sda.out‚ 861.44140625 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625* proc.memory.percent‚11.972570419311523 disk.rbd1.out‚107.0869140625 disk.drbd1965.in‚0 disk.drbd2041.in‚0% disk.nvme0n1.out‚53084.4267578125) disk./.usagePercent‚70.51404345831767 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0$ memory_percent‚92.13220278422037% disk./.usageGB‚1260.8627128601074 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1933.58984375 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd2118.out‚0â–FHÉ1‚(:½  ß‹Ó¶”‘´ disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0" disk.dm-0.out‚53084.6259765625 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 119.03125$ disk./.usageGB‚1260.862735748291 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚107.1884765625) disk./.usagePercent‚70.51404473834472 disk.nvme0n1.in‚ 119.03125) proc.memory.percent‚11.97792689005534 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0" proc.memory.rssMB‚ 2943.6953125 network.sent‚ 2.0502149e+07 network.recv‚ 1.526869e+06 disk.drbd1356.in‚0 disk.sda2.out‚ 861.765625 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 861.765625 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚53084.6259765625 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1931.88671875 cpu‚25.33894498862891 disk.drbd1552.in‚0$ memory_percent‚92.13913281758627 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚05l›Ê1‚(: ß‹Ó˜™ë³ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.552' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.85066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B€£^4Ë1/ _wandbruntime‚21385  _runtime‚21385äÂOñ;Ì1‹v 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.15312738716602325 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36490505933761597 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1870947927236557 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6672296524047852 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.669217586517334 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6459132432937622 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30127856135368347 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7169023752212524 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.693531334400177 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3868979513645172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8048428297042847 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2811810076236725 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22433356940746307 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5774756073951721 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.273817241191864 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.800389289855957 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09380018711090088 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7075346112251282 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.17219556868076324 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1700906902551651 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5078688263893127 +particle/sensitivity‚0.6425593495368958 +jet/accuracy_3_of_4‚0.10971427708864212 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8816624283790588 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.35224413871765137 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7848606223697538e+09 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3718503415584564 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6992206573486328 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.607692301273346 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.633414089679718 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 /particle/accuracy_4_of_4‚0.9671996831893921 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31273436546325684 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05462275817990303 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2720090448856354 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7198732495307922 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1751345843076706 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07612582296133041 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7124549746513367 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3941479027271271 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚7.725811537796316 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7232446074485779 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5442134141921997 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28237953782081604 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 epoch‚41 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40993431210517883 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7546698451042175 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.28347519040107727 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3901960551738739 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7627404928207397 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6600804924964905 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15785463154315948 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7251858711242676 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9218667149543762 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6901861429214478 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8737894296646118 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3608230650424957 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6402335166931152 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46302351355552673 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3612750172615051 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5951710939407349 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8817580938339233 *jet/accuracy_1_of_1‚0.3736281991004944 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4526953101158142 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6419752836227417 _runtime‚21385.727964998 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6560486555099487 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22086741030216217 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3608230650424957 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9526986479759216 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 6"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713583111763 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016619982197880745 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7028006315231323 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7307738065719604 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026092471554875374 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7765206098556519 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7248169183731079 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_2‚0.790306568145752 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3578384816646576 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8697951436042786 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.384472519159317 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6459132432937622 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8817580938339233 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3105030953884125 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27233925461769104 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8421576619148254 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45788004994392395 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04563154652714729 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029740314930677414 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.40717533230781555 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.33479851484298706 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1866709142923355 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016619982197880745 (particle/accuracy‚0.8554571270942688 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.37284067273139954 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 *jet/accuracy_2_of_3‚0.6056123971939087 *jet/accuracy_1_of_4‚0.9758731126785278 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8719929456710815 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8060821890830994 +jet/accuracy_4_of_4‚0.02295798622071743 trainer/global_step‚11759 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.374729186296463 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 +jet/accuracy_3_of_3‚0.07887604087591171 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *jet/accuracy_1_of_3‚0.9380072355270386 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6777617335319519 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6050236821174622 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6349537968635559 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9136089086532593 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03201308846473694 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6538852453231812 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5179873704910278 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 /particle/accuracy_3_of_3‚0.9993678331375122 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.29279831051826477 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29809334874153137 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8044556975364685 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7671446204185486 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6583966016769409 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02445843629539013 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7961539030075073 particle/accuracy_1_of_1‚1 _step‚981 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6342805027961731 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6987481117248535 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6406559348106384 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 }éj® Í1v 6"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713583111763 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016619982197880745 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7028006315231323 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7307738065719604 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026092471554875374 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7765206098556519 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7248169183731079 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_2‚0.790306568145752 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3578384816646576 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8697951436042786 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.384472519159317 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6459132432937622 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8817580938339233 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3105030953884125 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27233925461769104 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8421576619148254 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45788004994392395 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04563154652714729 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029740314930677414 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.40717533230781555 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.33479851484298706 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1866709142923355 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016619982197880745 QØOh/(particle/accuracy‚0.8554571270942688 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.37284067273139954 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 *jet/accuracy_2_of_3‚0.6056123971939087 *jet/accuracy_1_of_4‚0.9758731126785278 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8719929456710815 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8060821890830994 +jet/accuracy_4_of_4‚0.02295798622071743 trainer/global_step‚11759 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.374729186296463 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 +jet/accuracy_3_of_3‚0.07887604087591171 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *jet/accuracy_1_of_3‚0.9380072355270386 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6777617335319519 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6050236821174622 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6349537968635559 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9136089086532593 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03201308846473694 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6538852453231812 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5179873704910278 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 /particle/accuracy_3_of_3‚0.9993678331375122 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.29279831051826477 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29809334874153137 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8044556975364685 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7671446204185486 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6583966016769409 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02445843629539013 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7961539030075073 particle/accuracy_1_of_1‚1 _step‚981 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6342805027961731 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6987481117248535 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6406559348106384 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.15312738716602325 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36490505933761597 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1870947927236557 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6672296524047852 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.669217586517334 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6459132432937622 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30127856135368347 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7169023752212524 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.693531334400177 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3868979513645172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8048428297042847 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2811810076236725 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22433356940746307 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5774756073951721 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.273817241191864 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.800389289855957 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09380018711090088 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7075346112251282 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.17219556868076324 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1700906902551651 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5078688263893127 +particle/sensitivity‚0.6425593495368958 +jet/accuracy_3_of_4‚0.10971427708864212 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8816624283790588 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.35224413871765137 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7848606223697538e+09 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3718503415584564 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6992206573486328 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.607692301273346 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.633414089679718 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6326923370361328 /particle/accuracy_4_of_4‚0.9671996831893921 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31273436546325684 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05462275817990303 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2720090448856354 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7198732495307922 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1751345843076706 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07612582296133041 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7124549746513367 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3941479027271271 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚7.725811537796316 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7232446074485779 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5442134141921997 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28237953782081604 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 epoch‚41 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40993431210517883 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7546698451042175 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.28347519040107727 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3901960551738739 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7627404928207397 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6600804924964905 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15785463154315948 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7251858711242676 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9218667149543762 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6901861429214478 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8737894296646118 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3608230650424957 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6402335166931152 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46302351355552673 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3612750172615051 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5951710939407349 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8817580938339233 *jet/accuracy_1_of_1‚0.3736281991004944 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4526953101158142 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6419752836227417 _runtime‚21385.727964998 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6560486555099487 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22086741030216217 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3608230650424957 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9526986479759216 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301Õè*—›Î1‚(: ÒŸ‹Ó‘šÑ³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚296.218' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.73933333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áá£IÏ1‚(:¾  ÒŸ‹ÓÚØð³ disk.drbd2078.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2945.01171875 network.sent‚ 2.0502299e+07) disk./.usagePercent‚70.51404367165551 disk.drbd2041.out‚0% disk.nvme0n1.out‚53084.8974609375 disk.drbd1552.out‚0# memory_percent‚92.1454906463623 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.sda2.out‚ 862.15625 disk.dm-0.in‚ 119.03125" disk.dm-0.out‚53084.8974609375 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 cpu‚25.340190296485677 disk.drbd1938.in‚0 disk.drbd1356.in‚0) proc.memory.availableMB‚ 1930.32421875 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 119.03125 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.983283360799152 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚107.2509765625 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 862.15625% disk./.usageGB‚1260.8627166748047 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 network.recv‚ 1.527019e+06h\ËœÐ1‚(:’ ់Ó“ùµ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.453' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.15099999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅDScLÑ1‚(:Á  ់ÓÀ€Ø³ disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 862.5078125 disk.drbd1817.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1550.in‚0 disk.drbd1552.in‚0) disk./.usagePercent‚70.51405433854757 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚53085.17138671875 disk.drbd1965.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1510.in‚0 disk.drbd2118.in‚0# proc.memory.rssMB‚ 2945.01171875 network.recv‚ 1.528496e+06# disk.dm-0.out‚53085.17138671875 cpu‚25.340127665931707 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.sda.out‚ 862.5078125 disk.drbd1938.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.983283360799152 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚107.2509765625 disk.drbd1285.in‚0$ disk./.usageGB‚1260.862907409668 disk.drbd1828.in‚0 disk.drbd1510.out‚0& proc.memory.availableMB‚ 1930.15625 disk.drbd1931.in‚0$ memory_percent‚92.14617411295572 network.sent‚ 2.0562751e+07 disk.drbd2078.out‚0ÐÚc…LÒ1‚(:Á  ðŸ‹Óžã‡´ disk.sda1.in‚0 disk.sda.out‚ 862.77734375 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 network.recv‚ 1.52994e+06 disk.drbd1584.in‚0 disk.drbd1356.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1931.out‚0# disk.dm-0.out‚53085.83056640625 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.983283360799152 disk.drbd2118.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0& proc.memory.availableMB‚ 1930.15625 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 119.03125) disk./.usagePercent‚70.51405220516915 disk.sda1.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0$ memory_percent‚92.14617411295572 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 network.sent‚ 2.0621842e+07% disk./.usageGB‚1260.8628692626953 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.rbd1.out‚ 107.283203125# proc.memory.rssMB‚ 2945.01171875 cpu‚25.34007538492729 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 119.03125 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 862.77734375& disk.nvme0n1.out‚53085.83056640625 disk.drbd2078.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17ð¢‡EÓ1‚(:’ ðŸ‹ÓȈâ³ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚286.575' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.52499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.3ñ"œÔ1‚(:‘ ÿŸ‹Óâä³ gpu.0.gpu‚30.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.91' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.30333333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w«Z³LÕ1‚(:Á  ÿŸ‹ÓšÃƒ´# disk.dm-0.out‚53086.42041015625 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 network.sent‚ 2.0621908e+07 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 863.1171875 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1817.in‚0* proc.memory.percent‚11.983283360799152 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 network.recv‚ 1.530006e+06) disk./.usagePercent‚70.51406009866926 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 863.1171875# proc.memory.rssMB‚ 2945.01171875 disk.rbd1.in‚0 disk.rbd1.out‚ 107.384765625 disk.drbd2118.in‚0 disk.drbd1356.out‚0 cpu‚25.340035832316282& disk.nvme0n1.out‚53086.42041015625 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0& proc.memory.availableMB‚ 1930.15625 disk.drbd2041.out‚0% disk./.usageGB‚1260.8630104064941 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0$ memory_percent‚92.14617411295572 proc.cpu.threads‚17z 3=Ö1‚(:’ Ž ‹ÓŸíµ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚268.961' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.65366666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04 KûK×1‚(:À  Ž ‹Ó†§Ò³ disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 network.recv‚ 1.530928e+06 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2945.01171875 network.sent‚ 2.062495e+07 disk.drbd1817.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0# disk.dm-0.out‚53086.66748046875% disk./.usageGB‚1260.8629913330078 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 863.390625 disk.nvme0n1.in‚ 119.03125& disk.nvme0n1.out‚53086.66748046875 cpu‚25.339973909151816 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.sda.out‚ 863.390625 disk.dm-0.in‚ 119.03125 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0( proc.memory.availableMB‚ 1930.1484375 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51405903198005 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.983283360799152 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 107.447265625 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0$ memory_percent‚92.14620590209961©å4Ø1/ _wandbruntime‚21453  _runtime‚21453-—£eÇÙ1Á _runtime‚21453.710972875 _step‚982 % lr-AdamW/pg1‚7.28155407721285e-06 % lr-AdamW/pg2‚7.28155407721285e-06 trainer/global_step‚11799 % _timestamp‚1.7848606903537626e+09. ÈÌÚ1Æ _step‚982 % lr-AdamW/pg1‚7.28155407721285e-06 % lr-AdamW/pg2‚7.28155407721285e-06 trainer/global_step‚11799 % _timestamp‚1.7848606903537626e+09 _runtime‚21453.710972875Öì›`4Û1/ _wandbruntime‚21453  _runtime‚21453…T³§&Ü1’P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109588334 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212089556 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚944631556625408 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3398766108256256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3397425772055920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109598398 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3409722647809091 _step‚983 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3411083633289216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚944631135170048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚929568292910080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 _runtime‚21453.715892912 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109588334 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12317498859520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109598398 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3411082261679616 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102491158 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67423024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784860690354711e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚356 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31446167 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚929568661001216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12296875753171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3411083633289216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3398766134429696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12317498859520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3409721277769837 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212079486 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35977208 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15062895624192 trainer/global_step‚11799 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35976857 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3397425746627952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3409722647809091 _step‚983 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3411083633289216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚944631135170048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚929568292910080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 _runtime‚21453.715892912 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109588334 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12317498859520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109598398 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3411082261679616 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102491158 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67423024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784860690354711e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚356 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31446167 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚929568661001216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12296875753171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3411083633289216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3398766134429696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12317498859520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3409721277769837 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212079486 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35977208 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15062895624192 trainer/global_step‚11799 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35976857×B‘â+4Þ1/ _wandbruntime‚21455  _runtime‚21455J¸f0#'ß1N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12317175931392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35979834 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3411363565094912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3399045043720192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 _step‚984 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212107019 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102499519 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109597416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212096935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚929647247367168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12297896554923 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12296551943525 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12318521374720 trainer/global_step‚11799 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3399045017546752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3397704548401728 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3399045043720192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67428641 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109607494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31448812 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3397704573829696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109607494 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3410002470384619 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3410001100345253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚21455.174137453 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102499519 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12318521374720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67428989 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15064097151488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212107019 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672µãè('à1¢N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109607494 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3410002470384619 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3410001100345253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚21455.174137453 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102499519 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12318521374720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67428989 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 417253888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15064097151488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212107019 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12317175931392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35979834 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3411363565094912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3399045043720192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 _step‚984 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212107019 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102499519 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109597416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49162752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212096935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚929647247367168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12297896554923 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12296551943525 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12318521374720 trainer/global_step‚11799 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3399045017546752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3397704548401728 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3399045043720192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67428641 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109607494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31448812 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3397704573829696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328بÀ—é4á1/ _wandbruntime‚21455  _runtime‚21455‹¢øÜ â1‡ 2loss/SRqqt2/assignment_loss‚0.4886202555060884 4loss/FBt1/assignment_loss‚0.00016280117217234186 /loss/FRt2/detection_loss‚1.3788090015841667 1loss/SRqqt2/detection_loss‚0.5678442164139097 %loss/total_loss‚7.376565265956377 1loss/FBt2/assignment_loss‚0.03053578005707317 1loss/SRqqt1/detection_loss‚0.8155539047965852 0loss/FBt2/detection_loss‚0.32314099643758876 0loss/FRt1/assignment_loss‚1.4661462672200858 0loss/FRt2/assignment_loss‚0.6278522722182736 3loss/SRqqt1/assignment_loss‚0.46803929586248477 trainer/global_step‚11799 _runtime‚21455.219782984 /loss/FRt1/detection_loss‚0.5317686253072653 /loss/FBt1/detection_loss‚0.6780918493806836 epoch‚42 # _timestamp‚1.78486069186265e+09 _step‚985hEQ>ã1Œ /loss/FBt1/detection_loss‚0.6780918493806836 epoch‚42 # _timestamp‚1.78486069186265e+09 _step‚985 2loss/SRqqt2/assignment_loss‚0.4886202555060884 4loss/FBt1/assignment_loss‚0.00016280117217234186 /loss/FRt2/detection_loss‚1.3788090015841667 1loss/SRqqt2/detection_loss‚0.5678442164139097 %loss/total_loss‚7.376565265956377 1loss/FBt2/assignment_loss‚0.03053578005707317 1loss/SRqqt1/detection_loss‚0.8155539047965852 0loss/FBt2/detection_loss‚0.32314099643758876 0loss/FRt1/assignment_loss‚1.4661462672200858 0loss/FRt2/assignment_loss‚0.6278522722182736 3loss/SRqqt1/assignment_loss‚0.46803929586248477 trainer/global_step‚11799 _runtime‚21455.219782984 /loss/FRt1/detection_loss‚0.5317686253072653ÙZÜ8Eaä1‚(:‡  ‹Óñ•Ô³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.867' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.289 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoÔ–G†1ryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c› Oå1‚(:Ä   ‹Ó±©ú³ disk.drbd1510.out‚0) disk./.usagePercent‚70.51405604525029# disk.dm-0.out‚53086.89013671875 disk.drbd1226.in‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0( proc.memory.availableMB‚ 1930.3515625 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 network.sent‚ 2.0625016e+07 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚53086.89013671875 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 119.03125 disk.drbd1965.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 119.03125$ memory_percent‚92.14537938435873# proc.memory.rssMB‚ 2945.01171875$ disk./.usageGB‚1260.862937927246 disk.sda.in‚ 0.03515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 863.85546875 disk.drbd1828.in‚0 disk.rbd1.out‚ 107.447265625* proc.memory.percent‚11.983283360799152 network.recv‚ 1.530994e+06 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 cpu‚25.339933608394986 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 863.85546875 disk.drbd2078.out‚0göN¨Næ1‚(:à  ¬ ‹Óà¯í³ disk.drbd1285.out‚0$ disk./.usageGB‚1260.862937927246 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51405604525029 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0$ memory_percent‚92.14652379353842 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 cpu‚25.339871153816937 disk.rbd1.out‚107.4794921875 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 864.12109375 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 864.12109375 disk.drbd1931.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0" disk.dm-0.out‚53087.2802734375 network.sent‚ 2.0630658e+07 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 network.recv‚ 1.532045e+06 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.983283360799152 disk.sda2.in‚ 0.03515625# proc.memory.rssMB‚ 2945.01171875 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 119.03125( proc.memory.availableMB‚ 1930.0703125 disk.sda1.out‚0 disk.drbd1828.out‚0% disk.nvme0n1.out‚53087.2802734375 disk.drbd1673.out‚0óYf¢ç1‚(:’ ¬ ‹ÓåŒÌ³ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚295.501' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.50033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$Qðè1‚(:’ » ‹Óæ´â³ gpu.0.gpu‚99.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚284.101' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.70033333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08á´íPé1‚(:Å  » ‹Óá§ý³ disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0$ memory_percent‚92.14676221211752) proc.memory.availableMB‚ 1930.01171875 network.recv‚ 1.533785e+06 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 864.46484375 disk.drbd2078.out‚0% disk.nvme0n1.out‚53087.6513671875 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 119.03125" disk.dm-0.out‚53087.6513671875% disk./.usageGB‚1260.8631744384766 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0# proc.memory.rssMB‚ 2945.01171875 cpu‚25.339854681910428* proc.memory.percent‚11.983283360799152 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1965.out‚0 network.sent‚ 2.0693322e+07 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda2.out‚ 864.46484375) disk./.usagePercent‚70.51406927219642 disk.rbd1.out‚107.5810546875 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0q$€ê1‚(:’ Ê ‹Óê̳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚107.94' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.980000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/±1Eë1‚(:º  Ê ‹Óî¸ë³ disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0# disk.dm-0.out‚53087.95556640625 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0$ memory_percent‚92.14680989583334 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0% disk./.usageGB‚1260.8631744384766 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚53087.95556640625 disk.drbd1552.out‚0 proc.memory.availableMB‚1930 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚107.6435546875 disk.drbd1552.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.out‚0* proc.memory.percent‚11.983283360799152 disk.drbd1965.out‚0 network.recv‚ 1.534626e+06 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd2118.in‚0 cpu‚25.339804071379383) disk./.usagePercent‚70.51406927219642 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 864.828125 disk.drbd1673.out‚0 disk.sda2.out‚ 864.828125 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2945.01171875 network.sent‚ 2.0696321e+07 disk.dm-0.in‚ 119.03125ë3ì1‚(:’ Ù ‹Ó”ûå³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.597' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.86566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸKû Eí1‚(:º  Ù ‹Ó¬šƒ´ disk.drbd1226.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2945.01171875 network.sent‚ 2.0699429e+07 disk.dm-0.in‚ 119.03125 disk.drbd1550.in‚0* proc.memory.percent‚11.983283360799152 disk.rbd1.in‚0 disk.drbd1285.in‚0 network.recv‚ 1.535698e+06 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1584.in‚0 cpu‚25.33977592521896 disk.drbd1673.in‚0& disk.nvme0n1.out‚53088.14306640625 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0% disk./.usageGB‚1260.8633308410645 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.out‚865.375( proc.memory.availableMB‚ 1930.0546875 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0$ memory_percent‚92.14658737182617 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚107.6435546875 disk.drbd2118.out‚0( disk./.usagePercent‚70.5140780190479 disk.drbd1289.in‚0# disk.dm-0.out‚53088.14306640625 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚865.375 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0L¯Y•4î1/ _wandbruntime‚21539  _runtime‚21539dCúÈï1 & lr-AdamW/pg1‚6.961125152485575e-06 & lr-AdamW/pg2‚6.961125152485575e-06 trainer/global_step‚11849 % _timestamp‚1.7848607762427168e+09 _runtime‚21539.60090161 _step‚986ûj_.Íð1Ç % _timestamp‚1.7848607762427168e+09 _runtime‚21539.60090161 _step‚986 & lr-AdamW/pg1‚6.961125152485575e-06 & lr-AdamW/pg2‚6.961125152485575e-06 trainer/global_step‚11849ÚPýÄ|4ñ1/ _wandbruntime‚21539  _runtime‚215390JÖH(ò1“P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 trainer/global_step‚11849 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3425078915760640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚948623262918656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67703176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3423712470341445 % _timestamp‚1.7848607762435176e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚933497611207680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12368688415744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212953788 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚357 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12368688415744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110054350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36125361 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚948622843560448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12367342979584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _step‚987 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102909502 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3423713840380699 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3412711572781056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12347979635979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102909502 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31578172 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102909508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110044286 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67703533 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3411365860744720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12346635024693 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12367342979584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3425078915760640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3412711572781056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚933497979298816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110044286 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3425080287370240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102909508 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36125009 _runtime‚21539.601012877 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3412711598954496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212963858 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31578167 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15125283619840v™è¼(ó1˜P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67703176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3423712470341445 % _timestamp‚1.7848607762435176e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚933497611207680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12368688415744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212953788 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚357 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12368688415744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110054350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36125361 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚948622843560448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12367342979584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 _step‚987 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102909502 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3423713840380699 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3412711572781056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12347979635979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102909502 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31578172 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102909508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110044286 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67703533 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3411365860744720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12346635024693 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12367342979584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3425078915760640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3412711572781056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚933497979298816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110044286 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3425080287370240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102909508 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36125009 _runtime‚21539.601012877 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3412711598954496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212963858 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31578167 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15125283619840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 trainer/global_step‚11849 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3425078915760640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚948623262918656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181ÛDÉ¿“ô1‚(:ˆ è ‹Ó„ýÛ³ gpu.0.gpu‚39.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚303.234' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.078 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O.oøŸõ1‚(:Á  è ‹Ó­»ý³$ disk./.usageGB‚1260.863712310791 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 107.67578125 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0# memory_percent‚92.1487808227539 disk.drbd1226.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚53088.68310546875 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 network.sent‚ 2.0699495e+07 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51409935283198 disk.drbd2041.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 119.03125 disk.sda1.in‚0 network.recv‚ 1.535764e+06 disk.drbd2118.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.983283360799152& disk.nvme0n1.out‚53088.68310546875 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 865.58203125 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1356.out‚0 cpu‚25.339725195829875 disk.d’`ÿ(­rbd1510.out‚0 disk.sda.out‚ 865.58203125' proc.memory.availableMB‚ 1929.515625# proc.memory.rssMB‚ 2945.01171875 disk.drbd2118.out‚0 disk.drbd1356.in‚0©^f4ö1/ _wandbruntime‚21541  _runtime‚21541ÕÇ_o#'÷1N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67709065 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36127900 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚948702680109568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚933576565664768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102917875 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212981321 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚948703103654912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31580807 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31580812 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110063446 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12369710930944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12368365487616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚933576197573632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110053368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36128253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102917875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15126537990144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110053368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212981321 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3412990508244992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚11849 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12369710930944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102917869 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3411644637090528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67708707 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110063446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12347655826333 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212971237 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚21541.076893411 _step‚988 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212971237 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3425360219175936FGˆ­('ø1¢N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15126537990144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110053368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212981321 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3412990508244992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚11849 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12369710930944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102917869 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3411644637090528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67708707 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110063446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12347655826333 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212971237 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚353 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚21541.076893411 _step‚988 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212971237 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3425360219175936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67709065 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36127900 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚948702680109568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚933576565664768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102917875 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212981321 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚948703103654912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31580807 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31580812 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110063446 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12369710930944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12368365487616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚933576197573632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110053368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36128253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102917875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825ÜNW7š4ù1/ _wandbruntime‚21541  _runtime‚21541ñ[m– ú1‡ 0loss/FRt2/assignment_loss‚0.6161565610148885 2loss/SRqqt1/assignment_loss‚0.4981437670051435 3loss/SRqqt2/assignment_loss‚0.45599044321219623 _runtime‚21541.113140355 _step‚989 1loss/SRqqt1/detection_loss‚0.7537209557687793 /loss/FBt1/detection_loss‚0.7086063812096273 /loss/FBt2/detection_loss‚0.3299273256573995 trainer/global_step‚11849 % _timestamp‚1.7848607777559655e+09 2loss/FBt1/assignment_loss‚0.005724128756277021 2loss/FBt2/assignment_loss‚0.014387837036325218 /loss/FRt2/detection_loss‚1.3635986926052972 %loss/total_loss‚7.411112467029154 epoch‚42 0loss/FRt1/assignment_loss‚1.6195402418102298 /loss/FRt1/detection_loss‚0.5132634268665861 1loss/SRqqt2/detection_loss‚0.5320527060864051襔Óû1Œ 0loss/FRt2/assignment_loss‚0.6161565610148885 2loss/SRqqt1/assignment_loss‚0.4981437670051435 3loss/SRqqt2/assignment_loss‚0.45599044321219623 _runtime‚21541.113140355 _step‚989 1loss/SRqqt1/detection_loss‚0.7537209557687793 /loss/FBt1/detection_loss‚0.7086063812096273 /loss/FBt2/detection_loss‚0.3299273256573995 trainer/global_step‚11849 % _timestamp‚1.7848607777559655e+09 2loss/FBt1/assignment_loss‚0.005724128756277021 2loss/FBt2/assignment_loss‚0.014387837036325218 /loss/FRt2/detection_loss‚1.3635986926052972 %loss/total_loss‚7.411112467029154 epoch‚42 0loss/FRt1/assignment_loss‚1.6195402418102298 /loss/FRt1/detection_loss‚0.5132634268665861 1loss/SRqqt2/detection_loss‚0.5320527060864051ÝÜT×qžü1‚(:“ ÷ ‹Óˆðà³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚122.117' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.705666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Ñ-ŒKý1‚(:À  ÷ ‹ÓÐÏú³ proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚53088.91357421875* proc.memory.percent‚11.983283360799152 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0$ disk./.usageGB‚1260.863697052002 disk.drbd1965.out‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 cpu‚25.33966290658394 disk.drbd1938.in‚0 disk.rbd1.out‚ 107.77734375 disk.drbd1552.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd2078.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2945.01171875 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 network.recv‚ 1.538691e+06) disk./.usagePercent‚70.51409849948061 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 865.89453125 disk.drbd1965.in‚0 disk.drbd1817.in‚0# disk.dm-0.out‚53088.91357421875 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0& proc.memory.availableMB‚ 1929.96875 network.sent‚ 2.0765313e+07 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 865.89453125$ memory_percent‚92.14693705240884nâ<œþ1‚(:‘ †¡‹ÓˆÍª³ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.85' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.28333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[²NLÿ1‚(:Á  †¡‹Óž¶Ï³ network.recv‚ 1.539623e+06 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚53090.06005859375 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.983283360799152 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 866.203125 disk.dm-0.in‚ 119.03125 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1356.out‚0 network.sent‚ 2.0766333e+07 disk.drbd1828.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1929.72265625 disk.sda.out‚ 866.203125 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 107.83984375 cpu‚25.339647768607286 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0% disk./.usageGB‚1260.8635444641113 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0# disk.dm-0.out‚53090.06005859375$ memory_percent‚92.14793841044107 proc.cpu.threads‚17) disk./.usagePercent‚70.51408996596699 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 119.03125# proc.memory.rssMB‚ 2945.01171875Ðî×.€2‚(:’ •¡‹Ó¨˜ë³ gpu.0.gpu‚45.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚279.226' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.07533333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 PÈ%O2‚(:Ä  •¡‹Ó²µ…´ network.recv‚ 1.540616e+06( disk./.usagePercent‚70.5141036195888 disk.drbd2118.out‚0 disk.rbd1.out‚ 108.01953125 disk.nvme0n1.in‚ 119.03125 disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda.out‚ 866.69921875 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 network.sent‚ 2.0771419e+07 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1226.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2945.01171875 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.983283360799152 disk.drbd2078.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 866.69921875 disk.drbd1931.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.14793841044107 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚53090.25146484375 disk.drbd1510.in‚0 cpu‚25.339607661400443% disk./.usageGB‚1260.8637886047363 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1929.72265625# disk.dm-0.out‚53090.25146484375 disk.sda1.in‚0 disk.drbd1289.in‚0¦íö´N‚2‚(:à  ¤¡‹ÓÁšÑ³ disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.983283360799152 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 network.recv‚ 1.542407e+06# disk./.usageGB‚1260.86376953125 disk.drbd2118.out‚0 disk.sda.out‚ 866.95703125 disk.drbd1673.out‚0 disk.dm-0.in‚ 119.03125 disk.nvme0n1.in‚ 119.03125 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 108.08203125 disk.sda1.out‚0 cpu‚25.339546536843933 disk.drbd1356.out‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1929.72265625 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51410255289959 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚53090.97607421875 disk.drbd1550.in‚0 disk.sda1.in‚0$ memory_percent‚92.14793841044107 disk.drbd1673.in‚0 disk.sda2.out‚ 866.95703125 disk.drbd1356.in‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2945.01171875 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0 network.sent‚ 2.0772609e+07 disk.sda2.in‚ 0.03515625# disk.dm-0.out‚53090.97607421875z‡Ùúƒ2‚(:’ ¤¡‹Ó§Ù²³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.476' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.82533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Э“M„2‚(:  ³¡‹ÓêÉã³) disk./.usagePercent‚70.51410255289959 disk.drbd1552.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0# disk./.usageGB‚1260.86376953125 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 867.23828125 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1929.97265625 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.sda.out‚ 867.23828125 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1550.in‚0# disk.dm-0.out‚53091.42138671875 disk.rbd1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.14692115783691 disk.drbd1828.out‚0 network.recv‚ 1.543361e+06 disk.drbd2118.in‚0 disk.drbd2078.out‚0& disk.nvme0n1.out‚53091.42138671875 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2945.01171875 cpu‚25.33950722574096 disk.drbd1510.in‚0* proc.memory.percent‚11.983283360799152 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd2111.in‚0 network.sent‚ 2.0777636e+07 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.rbd1.out‚ 108.35546875z¢G…2‚(:’ ³¡‹Ó™¥Ç³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚254.521' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.84033333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒD;ÿ4†2/ _wandbruntime‚21626  _runtime‚21626_¥srȇ2 $ _timestamp‚1.784860863190047e+09 _runtime‚21626.547333053 _step‚990 & lr-AdamW/pg1‚6.647460249418861e-06 & lr-AdamW/pg2‚6.647460249418861e-06 trainer/global_step‚11899 ÜÓ͈2Ç & lr-AdamW/pg1‚6.647460249418861e-06 & lr-AdamW/pg2‚6.647460249418861e-06 trainer/global_step‚11899 $ _timestamp‚1.784860863190047e+09 _runtime‚21626.547333053 _step‚990ÞdR4‰2/ _wandbruntime‚21626  _runtime‚21626É>G3XŠ2’P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZFDòÚ&¶À'eviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚952614465297920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12397738907501 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103327852 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12418532535808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67982523 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 $ _timestamp‚1.784860863191176e+09 _step‚991 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36272356 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚952614890947584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15187593351168 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3426657063479296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12419877971968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12419877971968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103327858 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3425305949433520 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3426657037305856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12418532535808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67982892 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31710167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15187535792640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110510302 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110500238 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3439076941451264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3439075569841664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3439076941451264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3437705032952307 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181R±¶F(‹2—P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12419877971968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103327858 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3425305949433520 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3426657037305856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12418532535808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67982892 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31710167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15187535792640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110510302 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110500238 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3439076941451264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3439075569841664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3439076941451264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3437705032952307 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚952614465297920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12397738907501 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103327852 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12418532535808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67982523 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 $ _timestamp‚1.784860863191176e+09 _step‚991 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36272356 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚952614890947584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15187593351168 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3426657063479296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12419877971968 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213855623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3425584751207296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3426935972769792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36275682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3426935946596352 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12420900487168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3425584725779328 _runtime‚21628.023193982 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12419555043840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3426935946596352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12420900487168 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67988129 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31712812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31712807 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12419555043840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110519398 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110519398 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3437984855527835 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 trainer/global_step‚11899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213845539 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3437983485488469 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36275322 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110509320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 % _timestamp‚1.7848608646656115e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213855623 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3439356873256960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3426935972769792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110509320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15188780313088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103336219 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103336225 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3439355501640192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚952694719729664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825[ò{ÿ©Ž2¢N XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3437984855527835 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 trainer/global_step‚11899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213845539 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3437983485488469 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36275322 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110509320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 % _timestamp‚1.7848608646656115e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213855623 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3439356873256960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3426935972769792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110509320 MCDeviceStatsMonitor.on_train_batchO¶š2_end/allocation.small_pool.current‚10078 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15188780313088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103336219 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103336225 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3439355501640192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚952694719729664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12400104320539 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15188835767296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213855623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3425584751207296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3426935972769792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36275682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3426935946596352 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12420900487168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚365 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3425584725779328 _runtime‚21628.023193982 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12419555043840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3426935946596352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12420900487168 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67988129 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31712812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31712807 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12419555043840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110519398 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110519398 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224àz0æ42/ _wandbruntime‚21628  _runtime‚21628wçB 2‡ /loss/FRt2/detection_loss‚1.3218333766303054 0loss/SRqqt1/detection_loss‚0.904445192789925 1loss/FBt2/assignment_loss‚0.03532772992607674 4loss/FBt1/assignment_loss‚6.0379588357427434e-05 1loss/SRqqt2/detection_loss‚0.5583503170092257 0loss/FBt2/detection_loss‚0.29715905737848763 %loss/total_loss‚7.471082059223491 trainer/global_step‚11899 _runtime‚21628.063054307 0loss/FRt2/assignment_loss‚0.5882148972908314 2loss/SRqqt1/assignment_loss‚0.5058253452785039 /loss/FRt1/detection_loss‚0.5540155499786438 /loss/FBt1/detection_loss‚0.6280715652084088 epoch‚42 % _timestamp‚1.7848608647058008e+09 _step‚993 0loss/FRt1/assignment_loss‚1.6111380212516446 2loss/SRqqt2/assignment_loss‚0.4666406268930806êDÿ‘2Œ 1loss/FBt2/assignment_loss‚0.03532772992607674 4loss/FBt1/assignment_loss‚6.0379588357427434e-05 1loss/SRqqt2/detection_loss‚0.5583503170092257 0loss/FBt2/detection_loss‚0.29715905737848763 %loss/total_loss‚7.471082059223491 trainer/global_step‚11899 _runtime‚21628.063054307 0loss/FRt2/assignment_loss‚0.5882148972908314 2loss/SRqqt1/assignment_loss‚0.5058253452785039 /loss/FRt1/detection_loss‚0.5540155499786438 /loss/FBt1/detection_loss‚0.6280715652084088 epoch‚42 % _timestamp‚1.7848608647058008e+09 _step‚993 0loss/FRt1/assignment_loss‚1.6111380212516446 2loss/SRqqt2/assignment_loss‚0.4666406268930806 /loss/FRt2/detection_loss‚1.3218333766303054 0loss/SRqqt1/detection_loss‚0.904445192789925ážîW~’’2‚(:‡ ¡‹ÓÿæƳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.705' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.235 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž‹'N“2‚(:à  ¡‹Ó·Âæ³* proc.memory.percent‚11.983283360799152 network.sent‚ 2.0780674e+07 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0 disk.sda.out‚ 867.55859375 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 867.55859375 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd2078.in‚0% disk./.usageGB‚1260.8639030456543 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1673.in‚0 cpu‚25.339433658261278 disk.drbd1550.in‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2945.01171875% disk.nvme0n1.out‚53091.8974609375 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1510.out‚0 disk.drbd1965.out‚0$ memory_percent‚92.14170773824057 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1931.25390625 network.recv‚ 1.544337e+06) disk./.usagePercent‚70.51411001972401" disk.dm-0.out‚53091.8974609375 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.rbd1.out‚ 108.41796875˜Ê’”2‚(:‡ Ñ¡‹ÓìóÓ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚278.229' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.743 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|—ïH•2‚(:½  Ñ¡‹ÓÙÆò³ network.sent‚ 2.0840872e+07 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0% disk.nvme0n1.out‚53092.1865234375 disk.drbd1510.out‚0$ memory_percent‚92.14170773824057 cpu‚25.33939461040599 disk.drbd1550.out‚0) disk./.usagePercent‚70.51410895303482 disk.rbd1.out‚ 108.41796875 disk.nvme0n1.in‚ 119.03125$ disk./.usageGB‚1260.863883972168 disk.sda2.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.983283360799152 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2945.01171875 disk.dm-0.in‚ 119.03125 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 868.046875 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 network.recv‚ 1.545901e+06 disk.sda.in‚ 0.03515625 disk.sda.out‚ 868.046875 disk.drbd1673.out‚0" disk.dm-0.out‚53092.1865234375 disk.drbd2041.in‚0) proc.memory.availableMB‚ 1931.25390625 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0бîo–2‚(:’ à¡‹Ó‹¬¿³ gpu.0.gpu‚94.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚192.718' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.23933333333332 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Z L—2‚(:Á  à¡‹ÓŽÎß³$ memory_percent‚92.14170773824057 network.sent‚ 2.0843897e+07% disk./.usageGB‚1260.8601112365723 disk.drbd1938.out‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1931.25390625 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 868.234375 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0 disk.drbd2078.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚108.4501953125) disk./.usagePercent‚70.51389796191022 disk.drbd2118.out‚0 disk.drbd2111.in‚0 disk.sda1.out‚0 network.recv‚ 1.546944e+06 disk.drbd1817.out‚0 disk.sda.out‚ 868.234375% disk.nvme0n1.out‚53092.5341796875 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1828.out‚0# proc.memory.rssMB‚ 2945.01171875 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 cpu‚25.339356210978526 disk.drbd1289.in‚0" disk.dm-0.out‚53092.5341796875 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.983283360799152‘ ƒOO˜2‚(:Ä  ï¡‹ÓÎãݳ disk.drbd1289.in‚0 disk.drbd2111.in‚0% disk.nvme0n1.out‚53092.9521484375$ memory_percent‚92.14069048563638 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0" disk.dm-0.out‚53092.9521484375 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.983283360799152 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1931.50390625 disk.sda.in‚ 0.03515625 disk.drbd2041.out‚0) disk./.usagePercent‚70.51392569582953 disk.drbd1356.out‚0 network.sent‚ 2.0846898e+07 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 119.03125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 868.56640625 disk.drbd2111.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 868.56640625 disk.rbd1.out‚108.6376953125 disk.drbd1226.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8606071472168 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2945.01171875 cpu‚25.33929450297476 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 network.recv‚ 1.547892e+06 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd1673.out‚0¹Ë©ñž™2‚(:“ ï¡‹Óµ•½³ gpu.0.gpu‚80.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.671' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.223666666666674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ro´š2‚(:… þ¡‹Ó¹»³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.63' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.21 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0΄[çR›2‚(:Ç  þ¡‹ÓèþÚ³ disk.nvme0n1.in‚ 119.03125 disk.drbd1510.out‚0% disk./.usageGB‚1260.8606071472168 disk.drbd2041.in‚0 disk.drbd1673.in‚0# disk.dm-0.out‚53093.52001953125 disk.sda1.in‚0 disk.sda.out‚ 868.85546875 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0) proc.memory.availableMB‚ 1931.25390625$ memory_percent‚92.14170773824057 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 868.85546875 disk.drbd1931.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.983283360799152 network.recv‚ 1.548915e+06 disk.drbd1938.out‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.out‚0& disk.nvme0n1.out‚53093.52001953125# proc.memory.rssMB‚ 2945.01171875 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51392569582953 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 network.sent‚ 2.0849945e+07 disk.sda2.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚108.7001953125 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.sda1.out‚0 cpu‚25.339267419122763§`Šœ2‚(:’ ¢‹Óû‹µ³ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚274.013' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.33766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð_˜ªI2‚(:¾  ¢‹Ó§ïÕ³ disk.dm-0.in‚ 119.03125 disk.drbd2111.out‚0) proc.memory.availableMB‚ 1931.25390625 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 119.03125& disk.nvme0n1.out‚53093.72314453125 disk.drbd1510.out‚0) disk./.usagePercent‚70.51392569582953 disk.sda.in‚ 0.03515625 disk.sda.out‚869.3125 disk.drbd1673.in‚0 disk.sda2.out‚869.3125 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 cpu‚25.33922892214779 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1965.in‚0 network.recv‚ 1.549733e+06 disk.drbd2041.out‚0 network.sent‚ 2.0852939e+07 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.983283360799152 disk.rbd1.in‚0 disk.drbd1285.in‚0$ memory_percent‚92.14170773824057 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.rbd1.out‚108.7001953125 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2945.01171875% disk./.usageGB‚1260.8606071472168 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚53093.72314453125 disk.drbd1510.in‚0gxý4ž2/ _wandbruntime‚21712  _runtime‚21712˜¨æÉŸ2à & lr-AdamW/pg2‚6.340600482295053e-06 trainer/global_step‚11949 % _timestamp‚1.7848609494337084e+09 _runtime‚21712.791890734 _step‚994 & lr-AdamW/pg1‚6.340600482295053e-06”N1"Π2È trainer/global_step‚11949 % _timestamp‚1.7848609494337084e+09 _runtime‚21712.791890734 _step‚994 & lr-AdamW/pg1‚6.340600482295053e-06 & lr-AdamW/pg2‚6.340600482295053e-06â)ˆR4¡2/ _wandbruntime‚21712  _runtime‚21712Šæ Ó(¢2’P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚375 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110956190 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31842167 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3439246038122320 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110966254 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3451696225523915 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3453072223922688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3440602501830656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848609494344873e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3453073595532288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68262950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3440602501830656 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214712462 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚941356247802880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3453073595532288 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3453072223922688 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214702392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12450187401595 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12469722092032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103746202 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103746208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110956190 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12471067528192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 _runtime‚21712.79197965 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36420403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3440602528004096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3440602528004096f"~x%£2—P JDeviceStatsMonitor.on_train_batch_start/›€ÉÅø&allocation.all.current‚10070 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3440602501830656 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214712462 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚941356247802880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3453073595532288 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3453072223922688 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214702392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12450187401595 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12469722092032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103746202 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103746208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110956190 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12471067528192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 _runtime‚21712.79197965 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36420403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3440602528004096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3440602528004096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚375 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110956190 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31842167 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3439246038122320 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110966254 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3451696225523915 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 423552512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3453072223922688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3440602501830656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848609494344873e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3453073595532288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68262950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ã#¸ñ4¤2/ _wandbruntime‚21714  _runtime‚217149q#'¥2N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214729925 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36423379 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚941435202259968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103754575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12451208203347 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68268186 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12470744600064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110975350 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12470744600064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3439524814468128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103754575 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3451974678060077 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110975350 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110965272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31844807 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚11949 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110965272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12472090043392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3451976048099443 _runtime‚21714.270701204 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103754569 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68268575yû"('¦2¢N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3451974678060077 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110975350 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110965272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31844807 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 trainer/global_step‚11949 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110965272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12472090043392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 348207616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3451976048099443 _runtime‚21714.270701204 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103754569 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68268575 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214729925 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36423379 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚941435202259968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103754575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12451208203347 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68268186 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12470744600064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110975350 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12470744600064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1408391680 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3439524814468128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103754575 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400äתÜ4§2/ _wandbruntime‚21714  _runtime‚21714m §¨2ˆ # _timestamp‚1.78486095094727e+09 _step‚997 4loss/FBt1/assignment_loss‚1.1007889000570136e-05 1loss/SRqqt1/detection_loss‚0.8395852697994102 /loss/FBt1/detection_loss‚0.5031041378147322 trainer/global_step‚11949 _runtime‚21714.304461326 0loss/FRt2/assignment_loss‚0.5852502766109648 3loss/FBt2/assignment_loss‚0.0062516117114641235 0loss/FBt2/detection_loss‚0.34808020198804296 %loss/total_loss‚7.549859319255793 epoch‚42 2loss/SRqqt1/assignment_loss‚0.5813691332405738 3loss/SRqqt2/assignment_loss‚0.44053044953779613 /loss/FRt1/detection_loss‚0.5558904146642355 0loss/FRt1/assignment_loss‚1.6267677830664191 .loss/FRt2/detection_loss‚1.411669971591549 1loss/SRqqt2/detection_loss‚0.6513490613416045À§˜¹©2 3loss/FBt2/assignment_loss‚0.0062516117114641235 0loss/FBt2/detection_loss‚0.34808020198804296 %loss/total_loss‚7.549859319255793 epoch‚42 2loss/SRqqt1/assignment_loss‚0.5813691332405738 3loss/SRqqt2/assignment_loss‚0.44053044953779613 /loss/FRt1/detection_loss‚0.5558904146642355 0loss/FRt1/assignment_loss‚1.6267677830664191 .loss/FRt2/detection_loss‚1.411669971591549 1loss/SRqqt2/detection_loss‚0.6513490613416045 # _timestamp‚1.78486095094727e+09 _step‚997 4loss/FBt1/assignment_loss‚1.1007889000570136e-05 1loss/SRqqt1/detection_loss‚0.8395852697994102 /loss/FBt1/detection_loss‚0.5031041378147322 trainer/global_step‚11949 _runtime‚21714.304461326 0loss/FRt2/assignment_loss‚0.5852502766109648å¨Ï’üª2‚(:½  œ¢‹ÓýŽÙ³ disk.drbd2118.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 2945.01171875% disk./.usageGB‚1260.8644943237305 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0& proc.memory.availableMB‚ 1931.46875 network.sent‚ 2.085608e+07 network.recv‚ 1.550827e+06 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0# disk.dm-0.out‚53094.02685546875 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 869.640625 disk.drbd1673.out‚0 disk.rbd1.out‚ 108.732421875* proc.memory.percent‚11.983283360799152 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 disk.drØz7Lbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.in‚0) disk./.usagePercent‚70.51414308708935 disk.drbd1931.out‚0 disk.drbd1965.in‚0 cpu‚25.33917882975851 disk.sda1.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.14083353678384 disk.drbd1356.out‚0& disk.nvme0n1.out‚53094.02685546875 disk.drbd1510.in‚0 disk.nvme0n1.in‚ 119.03125 disk.sda2.out‚ 869.640625 disk.drbd1965.out‚0<½ÎÉœ«2‚(:‘ œ¢‹Óâª÷» gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚322.007' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚107.33566666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù:$»¬2‚(:’ «¢‹ÓËõá³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚272.565' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.85499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M‹#P­2‚(:Å  «¢‹Óµ™‚´ disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 869.99609375 disk.rbd1.out‚ 108.833984375 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 network.recv‚ 1.552699e+06 disk.drbd1828.out‚0& disk.nvme0n1.out‚53094.23779296875 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 869.99609375% disk./.usageGB‚1260.8644943237305 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.983283360799152 network.sent‚ 2.0915613e+07 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0) disk./.usagePercent‚70.51414308708935 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1930.20703125 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2945.01171875 cpu‚25.33913956899587 disk.sda.in‚ 0.03515625 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.14596748352051 disk.drbd1931.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚53094.23779296875 disk.drbd2041.in‚0¿4®2‚(:’ º¢‹ÓÙ÷˳ gpu.0.gpu‚87.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.917' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.30566666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M/ø¹F¯2‚(:»  º¢‹Ó¹›ë³ disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2945.01171875* proc.memory.percent‚11.983283360799152 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625$ memory_percent‚92.14693705240884 disk.drbd1285.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 network.sent‚ 2.0917499e+07 disk.sda.out‚ 870.23046875 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51414308708935 disk.dm-0.out‚53094.57421875 cpu‚25.33909009831541 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1356.in‚0 disk.drbd2111.out‚0& proc.memory.availableMB‚ 1929.96875 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚53094.57421875 disk.drbd1931.in‚0% disk./.usageGB‚1260.8644943237305 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 870.23046875 disk.rbd1.out‚ 108.896484375 network.recv‚ 1.55421e+06 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0wÂÖ’°2‚(:‡ É¢‹Ó÷쿳 gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚110.388' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09 ÖÖK±2‚(:À  É¢‹Ó—©á³) disk./.usagePercent‚70.51414308708935 disk.drbd1285.in‚0# disk.nvme0n1.out‚53094.77734375 disk.drbd1226.in‚0$ memory_percent‚92.14615821838379 network.recv‚ 1.562038e+06 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.sda.out‚ 870.72265625 disk.sda2.out‚ 870.72265625 disk.rbd1.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2945.01171875 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 network.sent‚ 2.0925933e+07 disk.drbd2118.in‚0 disk.dm-0.out‚53094.77734375 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8644943237305 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.983283360799152 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 cpu‚25.339051920163197 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1584.out‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1552.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0 disk.rbd1.out‚ 108.896484375 disk.nvme0n1.in‚ 119.03125) proc.memory.availableMB‚ 1930.16015625Í—)²2‚(:’ Ø¢‹Ó‡ªâ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.353' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.11766666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àáðO³2‚(:Ä  Ø¢‹Ó΂´ disk.drbd2078.out‚0 cpu‚25.338990061016194) disk./.usagePercent‚70.51414308708935 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0$ memory_percent‚92.14825630187988* proc.memory.percent‚11.983283360799152 disk.sda2.out‚ 870.98828125 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2945.01171875 proc.cpu.threads‚17% disk./.usageGB‚1260.8644943237305 disk.drbd1673.out‚0" disk.dm-0.out‚53095.1064453125 disk.drbd1828.in‚0 disk.rbd1.in‚0 network.recv‚ 1.563066e+06 disk.sda.out‚ 870.98828125 disk.drbd1938.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 108.896484375 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0) proc.memory.availableMB‚ 1929.64453125 disk.drbd1550.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚53095.1064453125 disk.drbd2118.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.in‚0 disk.drbd1289.in‚0 network.sent‚ 2.0928962e+07 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.dm-0.in‚ 119.03125 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625ö6 eœ´2‚(:‘ 碋ÓöÎä³ gpu.0.gpu‚20.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚292.924' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.64133333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|aêPµ2‚(:Å  碋Ó§¨…´ disk.drbd1817.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8645477294922 disk.dm-0.in‚ 119.03125 disk.drbd1828.out‚0% disk.nvme0n1.out‚53095.6923828125 disk.drbd1552.in‚0 disk.sda2.out‚ 871.30078125 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 network.sent‚ 2.0929769e+07 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 871.30078125 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51414607381913 disk.drbd2041.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.14727083841959 disk.drbd1817.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1929.88671875 disk.drbd2111.in‚0 disk.drbd1226.in‚0 cpu‚25.338974667637476 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0# proc.memory.rssMB‚ 2945.01171875 network.recv‚ 1.563198e+06 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1552.out‚0 disk.drbd1584.out‚0* proc.memory.percent‚11.983283360799152" disk.dm-0.out‚53095.6923828125 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚109.0302734375Yóä4¶2/ _wandbruntime‚21799  _runtime‚21799° *#È·2 & lr-AdamW/pg1‚6.040586073399134e-06 & lr-AdamW/pg2‚6.040586073399134e-06 trainer/global_step‚11999 $ _timestamp‚1.784861036459698e+09 _runtime‚21799.816937688 _step‚998ÿpN5͸2Ç _runtime‚21799.816937688 _step‚998 & lr-AdamW/pg1‚6.040586073399134e-06 & lr-AdamW/pg2‚6.040586073399134e-06 trainer/global_step‚11999 $ _timestamp‚1.784861036459698e+09æ¾Uš4¹2/ _wandbruntime‚21799  _runtime‚21799PÉL"(º2“P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 % _timestamp‚1.7848610364607785e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104164558 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31974172 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111412142 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3454547992528896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 _step‚999 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111422206 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12522257084416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68543244 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚397 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3453186126811120 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3467070249613312 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3467070249613312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215586764 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3467068878003712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12501291284403 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12520911648256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215576694 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3454547992528896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15311683425280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 trainer/global_step‚11999 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12522257084416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36568675 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12520911648256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚402 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68542842 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3465687418095523 _runtime‚21799.818268835 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104164552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31974167 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104164552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111422206 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 434038272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65947136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960597683564032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12499946673117 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111412142 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3467068878003712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36569072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚945285934191616 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚945285566100480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3453186101383152XæäZ(»2˜P H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12520911648256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215576694 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3454547992528896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15311683425280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 trainer/global_step‚11999 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12522257084416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36568675 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12520911648256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚402 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68542842 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1408391680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3465687418095523 _runtime‚21799.818268835 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104164552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31974167 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104164552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111422206 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 434038272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65947136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960597683564032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12499946673117 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111412142 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3467068878003712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36569072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚945285934191616 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚945285566100480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3453186101383152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 % _timestamp‚1.7848610364607785e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104164558 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 348207616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31974172 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111412142 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3454547992528896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 _step‚999 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111422206 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12522257084416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68543244 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚397 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3453186126811120 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3467070249613312 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3467070249613312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215586764 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3467068878003712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12501291284403 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejectioË;GÖ„%ns‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12521934156288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104172919 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3454826901819392 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 trainer/global_step‚11999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚402 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31976812 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15312915727360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12502312086155 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111421224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215604227 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3454826875645952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 % _timestamp‚1.7848610379339812e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12521934156288 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3467348809802240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15312981667328 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3453464903156928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215594143 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3467350181419008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68548842 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104172925 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12523279599616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚945364152466432 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36571633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚960677502224896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31976807 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3465965870631685 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68548440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚960677068193792 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3465967240671051 _step‚1000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3453464928584896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12500967474757 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111431302 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12523279599616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3467350181419008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 434031104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111431302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215604227 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3454826875645952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 % _timestamp‚1.7848610379339812e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12521934156288 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3467348809802240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15312981667328 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3453464903156928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215594143 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3467350181419008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68548842 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104172925 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12523279599616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚945364152466432 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36571633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚960677502224896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31976807 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3465965870631685 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68548440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚960677068193792 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3465967240671051 _step‚1000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3453464928584896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12500967474757 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111431302 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12523279599616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3467350181419008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 434031104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111431302 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12521934156288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104172919 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3454826901819392 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56èo’Sà4¿2/ _wandbruntime‚21801  _runtime‚21801h ±Ù À2… 0loss/FRt2/assignment_loss‚0.7117447302691002 1loss/SRqqt1/detection_loss‚0.8323132131597187 trainer/global_step‚11999 2loss/SRqqt2/assignment_loss‚0.5554880853569327 2loss/FBt2/assignment_loss‚0.014746030805548116 .loss/FRt1/detection_loss‚0.539451010229989 /loss/FBt2/detection_loss‚0.5312772729820572 epoch‚42 _step‚1001 0loss/FRt1/assignment_loss‚1.6812056692134734 2loss/SRqqt1/assignment_loss‚0.4819531840180178 .loss/FRt2/detection_loss‚1.362521726366528 0loss/SRqqt2/detection_loss‚0.567733554210196 /loss/FBt1/detection_loss‚0.4615029263051486 3loss/FBt1/assignment_loss‚0.0008411537496585659 %loss/total_loss‚7.740778556666369 % _timestamp‚1.7848610379725702e+09 _runtime‚21801.329826161aò?Á2Š 2loss/SRqqt2/assignment_loss‚0.5554880853569327 2loss/FBt2/assignment_loss‚0.014746030805548116 .loss/FRt1/detection_loss‚0.539451010229989 /loss/FBt2/detection_loss‚0.5312772729820572 epoch‚42 _step‚1001 0loss/FRt1/assignment_loss‚1.6812056692134734 2loss/SRqqt1/assignment_loss‚0.4819531840180178 .loss/FRt2/detection_loss‚1.362521726366528 0loss/SRqqt2/detection_loss‚0.567733554210196 /loss/FBt1/detection_loss‚0.4615029263051486 3loss/FBt1/assignment_loss‚0.0008411537496585659 %loss/total_loss‚7.740778556666369 % _timestamp‚1.7848610379725702e+09 _runtime‚21801.329826161 0loss/FRt2/assignment_loss‚0.7117447302691002 1loss/SRqqt1/detection_loss‚0.8323132131597187 trainer/global_step‚11999é? ÷-“Â2‚(:ˆ ö¢‹Ó×Ò¶³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚307.983' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.661 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VþEPÃ2‚(:Å  ö¢‹Ó©Ù³ disk.drbd1356.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.sda2.out‚ 871.51953125 disk.drbd2041.in‚0 disk.drbd2041.out‚0% disk./.usageGB‚1260.8645248413086 disk.drbd1226.in‚0 disk.rbd1.out‚109.0302734375 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1929.69921875 proc.cpu.threads‚17 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd2111.out‚0 network.sent‚ 2.0992146e+07 disk.drbd1552.out‚0 disk.sda.out‚ 871.51953125 disk.drbd1965.in‚0% disk.nvme0n1.out‚53096.1494140625) disk./.usagePercent‚70.51414479379207 disk.drbd2118.out‚0 disk.drbd1817.out‚0 cpu‚25.338925859597552 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 119.03125$ memory_percent‚92.14803377787271 disk.drbd1226.out‚0 disk.drbd1550.out‚0 network.recv‚ 1.565918e+06 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2945.01171875* proc.memory.percent‚11.983283360799152 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0" disk.dm-0.out‚53096.1494140625 disk.drbd1285.in‚0r–ÈÄ2‚(:’ …£‹Ó³ gpu.0.gpu‚90.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.189' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.72966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å[ ÀMÅ2‚(:  …£‹Óˆý˳ disk.drbd1938.in‚0% disk.nvme0n1.out‚53096.3681640625 network.recv‚ 1.566928e+06 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.out‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8645095825195 disk.drbd1550.out‚0" disk.dm-0.out‚53096.3681640625 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1929.65234375 disk.drbd1931.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625# proc.memory.rssMB‚ 2945.01171875 disk.sda.out‚ 872.16015625 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.rbd1.out‚109.0927734375" memory_percent‚92.148224512736 cpu‚25.338876513214668 disk.drbd1931.in‚0) disk./.usagePercent‚70.51414394044072 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 disk.nvme0n1.in‚ 119.03125 disk.sda2.out‚ 872.16015625 network.sent‚ 2.099514e+07 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.983283360799152¸ =^‘Æ2‚(:† ”£‹Óܯ³ gpu.0.gpu‚25.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.502' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.834 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Ý3‹HÇ2‚(:½  ”£‹ÓæŽ×³ disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 cpu‚25.338838216279385 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.drbd1226.in‚0 network.recv‚ 1.56706e+06$ disk.nvme0n1.out‚53097.076171875 disk.drbd1817.out‚0 disk.sda2.out‚ 872.390625 disk.rbd1.in‚0) proc.memory.availableMB‚ 1929.27734375# proc.memory.rssMB‚ 2945.01171875! disk.dm-0.out‚53097.076171875 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚109.0927734375 disk.drbd1550.out‚0 disk.sda.out‚ 872.390625 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0$ disk./.usageGB‚1260.865493774414 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 119.03125$ memory_percent‚92.14975039164224 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.983283360799152 disk.drbd1510.in‚0 disk.dm-0.in‚ 119.03125 network.sent‚ 2.0997971e+07) disk./.usagePercent‚70.51419898160366‹¦žÈ2‚(:“ ££‹Ó󟺳 gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.958' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.65266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­²Ö¿CÉ2‚(:¸  ££‹Óùþس disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 119.03125* proc.memory.percent‚11.983283360799152 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0! disk.dm-0.out‚53097.447265625 disk.rbd1.out‚ 109.2265625 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 872.65625 cpu‚25.33877730112117 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 872.65625( proc.memory.availableMB‚ 1929.1015625 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2945.01171875 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0$ memory_percent‚92.15046564737955 disk.drbd1510.out‚0 network.recv‚ 1.567168e+06% disk./.usageGB‚1260.8652420043945 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51418490130617 network.sent‚ 2.0998079e+07 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0$ disk.nvme0n1.out‚53097.447265625--ønÊ2‚(:’ ²£‹ÓôŸ¸³ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚273.215' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.07166666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏüqòIË2‚(:¾  ²£‹Ó½ÉÛ³ disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51418383461696 disk.drbd1965.out‚0 disk.rbd1.out‚ 109.2265625# proc.memory.rssMB‚ 2945.01171875 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 1931.0390625 network.sent‚ 2.0998145e+07% disk./.usageGB‚1260.8652229309082 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 cpu‚25.33876210027936 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.983283360799152 disk.drbd1584.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 119.03125 disk.sda.out‚ 873.01953125 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0! disk.dm-0.out‚53098.080078125$ disk.nvme0n1.out‚53098.080078125 network.recv‚ 1.567234e+06 disk.drbd2118.out‚0 disk.sda2.out‚ 873.01953125 disk.drbd1965.in‚0 proc.cpu.threads‚17$ memory_percent‚92.14258193969727 disk.sda1.in‚0 disk.drbd1550.out‚0šñìº(Ì2‚(:¹  Á£‹Ó®Öû³ disk.drbd1817.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 119.03125" proc.memory.rssMB‚ 2948.3359375 disk.drbd2118.out‚0 disk.sda.out‚ 873.6640625 disk.dm-0.in‚ 119.03125 disk.drbd1510.in‚0! disk.dm-0.out‚53098.521484375 disk.drbd1965.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1927.28515625 disk.drbd1938.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 873.6640625 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0( disk./.usagePercent‚70.5141908747657 cpu‚25.34025193903476 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0$ disk./.usageGB‚1260.865348815918 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚53098.521484375* proc.memory.percent‚11.996809641520183 disk.drbd1550.in‚0 disk.drbd1510.out‚0# memory_percent‚92.1578566233317 disk.sda.in‚ 0.035156ep™25 disk.rbd1.in‚0 disk.rbd1.out‚ 109.2890625 network.sent‚ 2.1001671e+07 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0 network.recv‚ 1.569206e+06 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0ª´Z„Í2‚(:’ Á£‹Óëàܳ gpu.0.gpu‚100.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚270.44' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.14666666666666 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QI4Î2/ _wandbruntime‚21889  _runtime‚21889@UÊu;Ï2Œv =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6622586250305176 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 )loss/val_total_loss‚7.719617529183073 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17532594501972198 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 particle/accuracy_1_of_1‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.16987843811511993 *jet/accuracy_1_of_1‚0.3722946047782898 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6951026320457458 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7115164399147034 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6340849995613098 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚12039 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3539328873157501 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35226917266845703 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6950607299804688 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6051632761955261 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04340752586722374 6"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8055459856987 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _step‚1002 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 /particle/accuracy_4_of_4‚0.9671996831893921 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 *jet/accuracy_1_of_4‚0.9794445633888245 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03082011267542839 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.40622684359550476 epoch‚42 $ _timestamp‚1.784861126282059e+09 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5510358214378357 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15771158039569855 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2677689790725708 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6959141492843628 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_1_of_2‚0.7897521257400513 (particle/accuracy‚0.8551257848739624 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.681483805179596 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6443090438842773 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03302428871393204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7193268537521362 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16995669901371002 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28753361105918884 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7565175294876099 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22362352907657623 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7219866514205933 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.753845751285553 $particle/f_score‚0.6556436419487 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02691933698952198 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3067553639411926 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9526986479759216 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38599640130996704 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1374969333410263 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34335970878601074 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29209521412849426 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6698874831199646 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.382292240858078 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8719929456710815 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27416640520095825 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6425356268882751 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05270404741168022 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025326119735836983 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3637879192829132 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8113698363304138 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6428655982017517 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6579282879829407 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6458988785743713 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29509589076042175 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30462077260017395 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.35443776845932007 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45552530884742737 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7756471037864685 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6501377820968628 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27717146277427673 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2830943167209625 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7693822979927063 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6501377820968628 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7163993120193481 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4594521224498749 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6134615540504456 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5241789221763611 _runtime‚21889.640096034 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36454659700393677 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.6081317067146301 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.366123229265213 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7808732390403748 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.883858859539032 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22713112831115723 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5952948331832886 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016619982197880745 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8039518594741821 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 *jet/accuracy_2_of_4‚0.8421576619148254 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38672420382499695 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6679516434669495 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7233352661132812 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35226917266845703 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35251638293266296 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26707273721694946 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07885482162237167 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8816624283790588 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6626723408699036 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6378545165061951 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746345639228821 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5141010880470276 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016619982197880745 *jet/accuracy_2_of_2‚0.1720188558101654 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8697951436042786 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9231768250465393 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4571056067943573 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18729279935359955 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7254734635353088 +jet/accuracy_3_of_4‚0.10971427708864212 +jet/accuracy_4_of_4‚0.02295798622071743 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9131055474281311 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8034043908119202 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40206319093704224 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06819179654121399 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7130255699157715 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6917619109153748 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.62973552942276 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3885233402252197 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.581583559513092 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08533710241317749 *jet/accuracy_1_of_3‚0.9408184885978699#$ÔÌ;Ð2‘v @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.681483805179596 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6443090438842773 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03302428871393204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7193268537521362 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16995669901371002 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28753361105918884 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7565175294876099 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22362352907657623 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7219866514205933 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.753845751285553 $particle/f_score‚0.6556436419487 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02691933698952198 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3067553639411926 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9526986479759216 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38599640130996704 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1374969333410263 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34335970878601074 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29209521412849426 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6698874831199646 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.382292240858078 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8719929456710815 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27416640520095825 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6425356268882751 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05270404741168022 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025326119735836983 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3637879192829132 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8113698363304138 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6428655982017517 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6579282879829407 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6458988785743713 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29509589076042175 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30462077260017395 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.35443776845932007 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45552530884742737 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7756471037864685 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6501377820968628 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27717146277427673 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2830943167209625 /particle/accuracy_3_of_3‚0.9993678331375122 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7693822979927063 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6501377820968628 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7163993120193481 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4594521224498749 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6134615540504456 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5241789221763611 _runtime‚21889.640096034 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36454659700393677 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.6081317067146301 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.366123229265213 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7808732390403748 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.883858859539032 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22713112831115723 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5952948331832886 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016619982197880745 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8039518594741821 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 *jet/accuracy_2_of_4‚0.8421576619148254 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38672420382499695 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6679516434669495 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7233352661132812 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35226917266845703 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35251638293266296 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26707273721694946 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07885482162237167 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8816624283790588 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6626723408699036 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6378545165061951 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746345639228821 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5141010880470276 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016619982197880745 *jet/accuracy_2_of_2‚0.1720188558101654 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8697951436042786 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9231768250465393 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4571056067943573 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18729279935359955 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7254734635353088 +jet/accuracy_3_of_4‚0.10971427708864212 +jet/accuracy_4_of_4‚0.02295798622071743 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9131055474281311 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8034043908119202 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40206319093704224 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06819179654121399 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7130255699157715 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6917619109153748 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.62973552942276 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3885233402252197 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.581583559513092 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08533710241317749 *jet/accuracy_1_of_3‚0.9408184885978699 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6622586250305176 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 )loss/val_total_loss‚7.719617529183073 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17532594501972198 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 particle/accuracy_1_of_1‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.16987843811511993 *jet/accuracy_1_of_1‚0.3722946047782898 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6951026320457458 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7115164399147034 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6340849995613098 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚12039 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3539328873157501 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35226917266845703 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6950607299804688 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6051632761955261 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04340752586722374 6"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8055459856987 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _step‚1002 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 /particle/accuracy_4_of_4‚0.9671996831893921 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 *jet/accuracy_1_of_4‚0.9794445633888245 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03082011267542839 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.40622684359550476 epoch‚42 $ _timestamp‚1.784861126282059e+09 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5510358214378357 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15771158039569855 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2677689790725708 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6959141492843628 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 *jet/accuracy_1_of_2‚0.7897521257400513 (particle/accuracy‚0.8551257848739624ê®D¦ÞGÑ2‚(:¼  У‹ÓˆÑ³ disk.drbd1817.out‚0* proc.memory.percent‚11.996809641520183 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 874.06640625 disk.sda2.in‚ 0.03515625" proc.memory.rssMB‚ 2948.3359375 network.sent‚ 2.1001737e+07 disk.drbd1226.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0) disk./.usagePercent‚70.51419002141434$ disk./.usageGB‚1260.865333557129 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.in‚0 cpu‚25.34081791457344 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0$ memory_percent‚92.15625127156576 disk.drbd1356.out‚0 disk.drbd1356.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 109.46875% disk.nvme0n1.out‚53099.2490234375 disk.drbd1550.out‚0 disk.drbd1931.in‚0 network.recv‚ 1.569272e+06 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0( proc.memory.availableMB‚ 1927.6796875" disk.dm-0.out‚53099.2490234375 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 874.06640625 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1289.out‚0 disk.drbd1289.in‚0{œhirÒ2‚(:’ У‹Ó寪³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* I p+gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚277.268' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.42266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'—j4Ó2/ _wandbruntime‚21905  _runtime‚21905‡¢õÚÈÔ2 _runtime‚ 21905.2671592 _step‚1003 & lr-AdamW/pg1‚5.747456347746454e-06 & lr-AdamW/pg2‚5.747456347746454e-06 trainer/global_step‚12049 % _timestamp‚1.7848611419099622e+09ìhËÍÕ2Ç _step‚1003 & lr-AdamW/pg1‚5.747456347746454e-06 & lr-AdamW/pg2‚5.747456347746454e-06 trainer/global_step‚12049 % _timestamp‚1.7848611419099622e+09 _runtime‚ 21905.2671592ëz¦2>4Ö2/ _wandbruntime‚21905  _runtime‚21905Mq^Ý(×2”P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104672340 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104672334 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36743848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12556659395234 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3471054107604992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12577719641088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3483633198855680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36744094 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 % _timestamp‚1.7848611419108565e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚964644239133184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3483631827246080 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15380389711872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3483633198855680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68856907 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104672334 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32113059 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216587735 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3482241850618354 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104672340 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚949263883911168 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3471054133778432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104672340 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚949263883911168 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3471054133778432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12577719641088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3483633198855680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36744094 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 % _timestamp‚1.7848611419108565e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚964644239133184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3483631827246080 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15380389711872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3483633198855680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68856907 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832ìÄ&L 4Ù2/ _wandbruntime‚21906  _runtime‚21906Û«G#'Ú2N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3483913130661376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104680701 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104680707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36746888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3469964018424864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12580087592448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12578742149120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15381705493504 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3483913130661376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68862854 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32115699 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111924483 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚949342470277120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104680707 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 _step‚1005 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111924483 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3471333043068928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111934561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36747150 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216615268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216605184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3482523043233136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12557680196874 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3471333016895488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3483911759044608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32115704 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3471333043068928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚964724139190784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404670976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216615268 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104680701 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68862587 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3482521673193770 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15381668913664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12578742149120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36579840 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 VGDeviceStatsMonitor.on_train_batch_end/reservÚ‡ãM&"ed_bytes.small_pool.current‚ 1730150400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104680707 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 _step‚1005 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111924483 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3471333043068928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111934561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36747150 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216615268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216605184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3482523043233136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12557680196874 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3471333016895488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3483911759044608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32115704 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3471333043068928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚964724139190784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404670976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216615268 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104680701 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68862587 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3482521673193770 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15381668913664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12578742149120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36579840 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3483913130661376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104680701 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104680707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36746888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3469964018424864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12580087592448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0í™kt4Ü2/ _wandbruntime‚21906  _runtime‚219068,¬—Ý2Š 0loss/FBt2/detection_loss‚0.45816431933444396 0loss/FRt2/assignment_loss‚0.6760698918738376 2loss/SRqqt2/assignment_loss‚0.4503186809801762 4loss/FBt1/assignment_loss‚7.3259438892905195e-06 2loss/FBt2/assignment_loss‚0.023096559077963322 0loss/FBt1/detection_loss‚0.46679305747101035 epoch‚43 trainer/global_step‚12049 _runtime‚21906.776174091 0loss/FRt1/assignment_loss‚1.5778690714510684 1loss/SRqqt1/assignment_loss‚0.524920568618983 /loss/FRt1/detection_loss‚0.5347461510619741 % _timestamp‚1.7848611434190896e+09 %loss/total_loss‚7.575103808804375 _step‚1006 /loss/FRt2/detection_loss‚1.3880838869051149 1loss/SRqqt1/detection_loss‚0.8643264844264704 1loss/SRqqt2/detection_loss‚0.6107078116594444Ç ŒÞ2 2loss/SRqqt2/assignment_loss‚0.4503186809801762 4loss/FBt1/assignment_loss‚7.3259438892905195e-06 2loss/FBt2/assignment_loss‚0.023096559077963322 0loss/FBt1/detection_loss‚0.46679305747101035 epoch‚43 trainer/global_step‚12049 _runtime‚21906.776174091 0loss/FRt1/assignment_loss‚1.5778690714510684 1loss/SRqqt1/assignment_loss‚0.524920568618983 /loss/FRt1/detection_loss‚0.5347461510619741 % _timestamp‚1.7848611434190896e+09 %loss/total_loss‚7.575103808804375 _step‚1006 /loss/FRt2/detection_loss‚1.3880838869051149 1loss/SRqqt1/detection_loss‚0.8643264844264704 1loss/SRqqt2/detection_loss‚0.6107078116594444 0loss/FBt2/detection_loss‚0.45816431933444396 0loss/FRt2/assignment_loss‚0.6760698918738376îÍ{UŒß2‚(:’ ߣ‹ÓÁþ¹³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.797' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.93233333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U5Eà2‚(:º  ߣ‹Óʬܳ disk.drbd1931.out‚0$ memory_percent‚92.15755462646484 disk.rbd1.out‚ 109.8046875% disk.nvme0n1.out‚53099.4873046875 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0" disk.dm-0.out‚53099.4873046875 disk.drbd1965.in‚0 network.recv‚ 1.570427e+06 disk.drbd1584.out‚0 disk.sda.out‚ 874.484375 disk.drbd1289.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51419002141434 disk.drbd1550.in‚0 disk.drbd1285.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.996809641520183 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0' proc.memory.availableMB‚ 1927.359375" proc.memory.rssMB‚ 2948.3359375 cpu‚25.340778414100846 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2118.out‚0 network.sent‚ 2.1009941e+07 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 119.03125$ disk./.usageGB‚1260.865333557129 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 874.484375¬4é¼á2‚(:’ Óß»³ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚285.311' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.10366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þpDqDâ2‚(:¹  Óæ׳ disk.drbd1673.in‚0 disk.dm-0.out‚53099.90234375 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1552.in‚0 disk.drbd2111.in‚0' proc.memory.availableMB‚ 1927.859375 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0# memory_percent‚92.1555201212565" proc.memory.rssMB‚ 2948.3359375 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 119.03125 disk.drbd1510.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51419002141434$ disk./.usageGB‚1260.865333557129 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 cpu‚25.340727898080072 disk.sda.out‚ 874.73828125 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 109.8046875 network.sent‚ 2.1010049e+07 disk.drbd1289.out‚0# disk.nvme0n1.out‚53099.90234375 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 874.73828125 disk.drbd1931.out‚0* proc.memory.percent‚11.996809641520183 disk.drbd1289.in‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 119.03125 disk.rbd1.in‚0 network.recv‚ 1.570535e+06 disk.drbd1550.out‚0r)©áœã2‚(:‘ ý£‹Ó·«Ã³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚307.293' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.43100000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý a@ä2‚(:µ  ý£‹Ó€‹ã³ disk.drbd1285.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.15819040934244* proc.memory.percent‚11.996809641520183 disk.drbd2118.out‚0 disk.drbd1510.out‚0 network.recv‚ 1.574515e+06 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0 cpu‚25.34068850643846 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.sda.out‚ 875.140625 disk.sda2.in‚ 0.03515625 disk.dm-0.out‚53100.09765625 disk.drbd1356.in‚0# disk.nvme0n1.out‚53100.09765625 network.sent‚ 2.1093538e+07) disk./.usagePercent‚70.51419002141434 disk.sda2.out‚ 875.140625 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0$ disk./.usageGB‚1260.865333557129" proc.memory.rssMB‚ 2948.3359375 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 proc.cpu.threads‚17 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 109.8828125 disk.drbd1285.out‚0' proc.memory.availableMB‚ 1927.203125 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.rbd1.in‚0nÉ…å2‚(:’ Œ¤‹Óó•É³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.513' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.83766666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šµŽ…Gæ2‚(:¼  Œ¤‹Ó–Õé³ disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 875.47265625 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 network.sent‚ 2.1094616e+07 network.recv‚ 1.57556e+06 disk.drbd1285.in‚0& disk.nvme0n1.out‚53100.51123046875 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.15876261393228 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1226.out‚0" proc.memory.rssMB‚ 2948.3359375$ disk./.usageGB‚1260.865333557129 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 cpu‚25.34061516903092) disk./.usagePercent‚70.51419002141434 disk.drbd1584.in‚0 disk.sda2.out‚ 875.47265625 disk.drbd1931.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.996809641520183 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0# disk.dm-0.out‚53100.51123046875 disk.drbd1965.in‚0 disk.dm-0.in‚ 119.03125 disk.nvme0n1.in‚ 119.03125 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 109.8828125% proc.memory.availableMB‚ 1927.0625?<~|Iç2‚(:¾  ›¤‹Óƒ¨‡´ proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.rbd1.in‚0 cpu‚25.340575575694476 disk.drbd1550.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1938.in‚0 disk.sda2.out‚ 875.8203125 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚53100.95263671875 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0# disk.dm-0.out‚53100.95263671875 disk.drbd1356.in‚0 disk.drbd2041.out‚0 network.sent‚ 2.1097064e+07 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.drbd1356.out‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.51419002141434 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚11.996809641520183 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0$ memory_percent‚92.15850830078125" proc.memory.rssMB‚ 2948.3359375 disk.sda2.in‚ 0.03515625 disk.drbd1584.in‚0 disk.drbd1673.in‚0$ disk./.usageGB‚1260.865333557129 disk.sda.out‚ 875.8203125 disk.dm-0.in‚ 119.03125 disk.drbd1828.out‚0 network.recv‚ 1.576397e+06 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 119.03125$ proc.memory.availableMB‚1927.125 disk.drbd1226.out‚0 disk.drbd2078.out‚0 disk.rbd1.out‚110.0166015625 disk.drbd1285.out‚0_oDè2‚(:’ ›¤‹ÓòÒá³ gpu.0.gpu‚87.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.033' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.01100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Éé2‚(:’ ª¤‹Ó‰Øã³ gpu.0.gpu‚20.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚318.788' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚106.26266666666666 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&C€[Kê2‚(:À  ª¤‹ÓÁ«…´ disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0" proc.memory.rssMB‚ 2948.3359375 cpu‚25.340525267225317 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 network.sent‚ 2.1100101e+07 disk.drbd1552.in‚0 disk.drbd1356.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚110.0166015625 disk.sda.in‚ 0.03515625 disk.drbd1552.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51419002141434 disk.dm-0.in‚ 119.03125( proc.memory.availableMB‚ 1926.6953125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 119.03125# disk.dm-0.out‚53101.41552734375 disk.drbd1965.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 876.109375* proc.memory.percent‚11.996809641520183 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1550.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚53101.41552734375 disk.drbd1226.in‚0 disk.sda2.out‚ 876.109375 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0$ memory_percent‚92.16025670369467 network.recv‚ 1.577497e+06 disk.drbd2041.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚1260.865333557129Ü/vÙ4ë2/ _wandbruntime‚21991  _runtime‚21991êºM]Éì2à & lr-AdamW/pg1‚5.461249727928064e-06 & lr-AdamW/pg2‚5.461249727928064e-06 trainer/global_step‚12099 $ _timestamp‚1.784861228110371e+09 _runtime‚21991.467596076 _step‚1007çn¿Îí2È _runtime‚21991.467596076 _step‚1007 & lr-AdamW/pg1‚5.461249727928064e-06 & lr-AdamW/pg2‚5.461249727928064e-06 trainer/global_step‚12099 $ _timestamp‚1.784861228110371e+09ïQ4î2/ _wandbruntime‚21991  _runtime‚21991¼d„ÿž%ï2”P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3497628481327104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36893446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚953193570299904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12628909197312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚968636716948992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12630254633472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3484999572129792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69138168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15443559715840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105090690 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3497629852936704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12607763278042 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3497628481327104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15443514740224 % _timestamp‚1.7848612281114388e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105090684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12628909197312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217462037 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3484999598303232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3484999598303232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217472107 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 trainer/global_step‚12099 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚968637130015744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatU{'~|sMonitor.on_train_batch_start/segment.small_pool.current‚825 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3483625279911920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864(1}(ð2™P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69138510 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3484999572129792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69138168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15443559715840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44975616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105090690 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3497629852936704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12607763278042 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3497628481327104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15443514740224 % _timestamp‚1.7848612281114388e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105090684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12628909197312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217462037 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3484999598303232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3484999598303232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217472107 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 trainer/global_step‚12099 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚968637130015744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3483625279911920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚337 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3497628481327104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36893446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚953193570299904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12628909197312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚968636716948992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12630254633472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32247699 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217489570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚342 $ _timestamp‚1.784861229588527e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3485278481420288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105099057 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105099051 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3497909784742400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217479486 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105099057 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32247704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚953272156665856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15444813495296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112380435 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3497909784742400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚968716550810112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3483904107113664 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12608784079682 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15444762235392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105099051 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112390513 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36896396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112380435 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3497908413125632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚953271788574720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12631277148672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36896054 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3496512865765378 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12629931705344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217489570 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12629931705344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 trainer/global_step‚12099 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69143753 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12629931705344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217489570 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12629931705344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 trainer/global_step‚12099 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69143753 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32247699 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217489570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚342 $ _timestamp‚1.784861229588527e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3485278481420288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105099057 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105099051 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3497909784742400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217479486 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105099057 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32247704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚953272156665856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15444813495296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112380435 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3497909784742400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚968716550810112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3483904107113664 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12608784079682 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15444762235392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105099051 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112390513 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36896396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112380435 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768ñ™®üÒ4ô2/ _wandbruntime‚21992  _runtime‚21992ùÕo õ2† 1loss/FBt1/assignment_loss‚0.01187250708346242 .loss/FRt1/detection_loss‚0.546184655832277 /loss/FRt2/detection_loss‚1.3925117843951367 0loss/FBt2/detection_loss‚0.48830962585452653 _step‚1010 0loss/FRt1/assignment_loss‚1.5937965578568263 0loss/FRt2/assignment_loss‚0.6310125149969004 2loss/FBt2/assignment_loss‚0.023417270213428883 1loss/SRqqt2/detection_loss‚0.5685150874390578 .loss/FBt1/detection_loss‚0.493702372820003 trainer/global_step‚12099 _runtime‚21992.983214126 3loss/SRqqt1/assignment_loss‚0.47772447530917056 2loss/SRqqt2/assignment_loss‚0.4898103244356364 % _timestamp‚1.7848612296260078e+09 1loss/SRqqt1/detection_loss‚0.8975222449611269 %loss/total_loss‚7.614379421197553 epoch‚43…™:(ö2‹ 0loss/FRt1/assignment_loss‚1.5937965578568263 0loss/FRt2/assignment_loss‚0.6310125149969004 2loss/FBt2/assignment_loss‚0.023417270213428883 1loss/SRqqt2/detection_loss‚0.5685150874390578 .loss/FBt1/detection_loss‚0.493702372820003 trainer/global_step‚12099 _runtime‚21992.983214126 3loss/SRqqt1/assignment_loss‚0.47772447530917056 2loss/SRqqt2/assignment_loss‚0.4898103244356364 % _timestamp‚1.7848612296260078e+09 1loss/SRqqt1/detection_loss‚0.8975222449611269 %loss/total_loss‚7.614379421197553 epoch‚43 1loss/FBt1/assignment_loss‚0.01187250708346242 .loss/FRt1/detection_loss‚0.546184655832277 /loss/FRt2/detection_loss‚1.3925117843951367 0loss/FBt2/detection_loss‚0.48830962585452653 _step‚1010ò.ßô V÷2‚(:’ ¹¤‹ÓÐÓ̳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllo#ÍyGcated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.318' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.77266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&çéPø2‚(:Å  ¹¤‹Óýçï³ disk.drbd1552.out‚0 disk.drbd1673.out‚0# disk.dm-0.out‚53101.79833984375 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 network.sent‚ 2.1103099e+07 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.996809641520183 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 network.recv‚ 1.578482e+06& disk.nvme0n1.out‚53101.79833984375% disk./.usageGB‚1260.8672790527344 disk.drbd2118.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 119.03125 disk.sda2.out‚ 876.54296875 disk.rbd1.in‚0 cpu‚25.340475397717487 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1285.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.15834935506186( proc.memory.availableMB‚ 1927.1640625 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚110.0791015625 disk.drbd1226.in‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2948.3359375 proc.cpu.threads‚17) disk./.usagePercent‚70.51429882371318 disk.dm-0.in‚ 119.03125 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1510.out‚0 disk.sda.out‚ 876.54296875 disk.drbd1965.in‚0­Õÿk’ù2‚(:‡ Ȥ‹Óý©â³ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚249.093' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.031 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#òtêMú2‚(:  Ȥ‹Ó‹˜ý³ disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 cpu‚25.340458952847943 disk.dm-0.in‚ 119.03125# disk.dm-0.out‚53102.19189453125 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 1926.8828125 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚110.0791015625 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625" proc.memory.rssMB‚ 2948.3359375 network.sent‚ 2.1163109e+07 disk.drbd1285.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚53102.19189453125 disk.drbd1817.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0$ memory_percent‚92.15949376424155 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 876.8203125% disk./.usageGB‚1260.8723335266113 disk.drbd1584.in‚0* proc.memory.percent‚11.996809641520183 network.recv‚ 1.580179e+06 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.sda.out‚ 876.8203125 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625( disk./.usagePercent‚70.5145814963523 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0¸I©‹Kû2‚(:À  פ‹ÓùÜó³ proc.cpu.threads‚17 network.sent‚ 2.1166156e+07 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.out‚0 disk.drbd2041.out‚0$ disk./.usageGB‚1260.872314453125& disk.nvme0n1.out‚53102.64111328125 disk.drbd1817.out‚0# disk.dm-0.out‚53102.64111328125 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 network.recv‚ 1.5812e+06 disk.sda.out‚ 877.01171875 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd2041.in‚0" proc.memory.rssMB‚ 2948.3359375 cpu‚25.340420326764544 disk.dm-0.in‚ 119.03125 disk.drbd1356.in‚0( proc.memory.availableMB‚ 1926.8828125* proc.memory.percent‚11.996809641520183( disk./.usagePercent‚70.5145804296631 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 877.01171875 disk.drbd1289.out‚0 disk.rbd1.out‚ 110.212890625 disk.nvme0n1.in‚ 119.03125$ memory_percent‚92.15949376424155кS—›ü2‚(: פ‹ÓöÅÒ³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚220.443' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.48100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ˜õý2‚(:’ 椋Ó¸Ïá³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚66.0 gpu.0.powerWatts‚293.404' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.80133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äxf+Nþ2‚(:à  椋Óêøÿ³ network.sent‚ 2.1168976e+07 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 cpu‚25.340358655892672 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0$ memory_percent‚92.15847651163736# disk.dm-0.out‚53103.64013671875 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.sda.out‚ 877.30078125 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0" proc.memory.rssMB‚ 2948.3359375 disk.sda2.out‚ 877.30078125 disk.drbd1931.out‚0 disk.rbd1.out‚ 110.212890625 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.996809641520183 disk.sda2.in‚ 0.03515625 disk.drbd2041.out‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0( proc.memory.availableMB‚ 1927.1328125) disk./.usagePercent‚70.51438437218735$ disk./.usageGB‚1260.868808746338 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 119.03125& disk.nvme0n1.out‚53103.64013671875 network.recv‚ 1.581398e+06 disk.drbd1965.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0Wcé ÿ2‚(:’ õ¤‹Óظس gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.863' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.28766666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;{.éG€3‚(:¼  õ¤‹Ó˜÷³# disk.dm-0.out‚53103.84326171875 disk.drbd1552.in‚0 disk.drbd1938.in‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.996809641520183 cpu‚25.340320110014844 disk.drbd1552.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0$ memory_percent‚92.15949376424155 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625" proc.memory.rssMB‚ 2948.3359375 network.sent‚ 2.1169042e+07 disk.sda2.out‚877.6875 disk.dm-0.in‚ 119.03125 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 network.recv‚ 1.581464e+06 disk.drbd2078.out‚0) disk./.usagePercent‚70.51437989209269 disk.drbd2118.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.rbd1.out‚ 110.275390625 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚877.6875 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚53103.84326171875 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 119.03125 proc.cpu.threads‚17% disk./.usageGB‚1260.8687286376953 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 1926.8828125 disk.drbd1356.in‚0 disk.drbd2041.out‚0t§DÁ43/ _wandbruntime‚22078  _runtime‚22078Âå-É‚3à & lr-AdamW/pg1‚5.182003729074541e-06 & lr-AdamW/pg2‚5.182003729074541e-06 trainer/global_step‚12149 $ _timestamp‚1.784861315064365e+09 _runtime‚22078.421588954 _step‚1011Ñâ&¹Îƒ3È & lr-AdamW/pg1‚5.182003729074541e-06 & lr-AdamW/pg2‚5.182003729074541e-06 trainer/global_step‚12149 $ _timestamp‚1.784861315064365e+09 _runtime‚22078.421588954 _step‚1011ó)–c•4„3/ _wandbruntime‚22078  _runtime‚22078Ò¬„Ô(…3”P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚972628301298176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3511625135408128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3511625135408128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218336339 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3498945036654592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12681444189696 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112837369 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3510224235761570 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3498945062828032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15505832058880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848613150653875e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚371 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218346409 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37041242 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105509040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105509034 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37040876 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚22078.422932832 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3511626507017728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12681444189696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15505780791808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12680098753536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 _step‚1012 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3498945062828032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3498945036654592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32377064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32377059 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160Ë –(†3™P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218336339 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3498945036654592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12681444189696 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112837369 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3510224235761570 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3498945062828032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15505832058880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7848613150653875e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚371 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218346409 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37041242 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105509040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105509034 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37040876 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚22078.422932832 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3511626507017728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12681444189696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15505780791808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12680098753536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 _step‚1012 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3498945062828032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3498945036654592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32377064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32377059 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚972628301298176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3511625135408128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3511625135408128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070ôÕ7Œø4‡3/ _wandbruntime‚22079  _runtime‚22079Ú<äŒëˆ3œN E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚972708561920000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218363872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3511906438823424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105517401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32379699 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚379 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112836387 ;4DeviceStatsMoniDyÂy7!tor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12661232573888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚957201474963456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3499223945945088 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69423514 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚12149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12682466704896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚957201106872320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 $ _timestamp‚1.784861316537779e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69423893 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218353788 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12682466704896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105517401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112846465 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112836387 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218353788 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3499223945945088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3510505428376352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105517407 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15507031502336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12681121261568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37044189 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3499223972118528 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3497844170374496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3499223972118528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3511906438823424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15507086956544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3497844195802464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218363872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37043815 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3510504058336986 _runtime‚22079.89533355 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112846465 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32379704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¿C?I''‰3¡N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚972708138374656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69423514 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 trainer/global_step‚12149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12682466704896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚957201106872320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 $ _timestamp‚1.784861316537779e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69423893 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218353788 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12682466704896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105517401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112846465 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112836387 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218353788 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3499223945945088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3510505428376352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105517407 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15507031502336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12681121261568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37044189 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3499223972118528 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3497844170374496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3499223972118528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3511906438823424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15507086956544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3497844195802464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218363872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37043815 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3510504058336986 _runtime‚22079.89533355 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112846465 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32379704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚972708561920000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218363872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3511906438823424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105517401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32379699 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚379 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112836387 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12661232573888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚957201474963456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3499223945945088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3511505482717488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3525621789489152 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105927384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12732633745920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3524215428333178 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3511505457289520 _runtime‚22164.208925891 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚976620518696960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69698204 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219220711 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12711315654944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚377 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15568311801856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12709971043658 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37189145 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37189145 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3511505482717488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3525621789489152 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105927384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12732633745920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3524215428333178 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3511505457289520 _runtime‚22164.208925891 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚976620518696960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69698204 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181øw! 4Ÿ3/ _wandbruntime‚22165  _runtime‚22165 çÛ€$' 3žN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚22165.681235614 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _step‚1017 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12733656261120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3524495250908594 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105935757 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219238174 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3511784284491264 % _timestamp‚1.7848614023236551e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12710991845298 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3513169410469888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69703746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚976700347401216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113302417 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32511699 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3524496620947960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12732310817792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12712336456696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219228090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113292339 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69704118 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37192414 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32511704 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113292339 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚961130793261056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12732310817792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15569496588800 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113302417 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3511784259063296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105935751 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3513169436643328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3525903092904448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15569554140160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219238174 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚372 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105935757 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264Äq%¡3£N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113292339 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚961130793261056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12732310817792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15569496588800 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113302417 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3511784259063296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105935751 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3513169436643328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3525903092904448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15569554140160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated,©­Œ‚ 219238174 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚372 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105935757 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚22165.681235614 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _step‚1017 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12733656261120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3524495250908594 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105935757 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219238174 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3511784284491264 % _timestamp‚1.7848614023236551e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12710991845298 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3513169410469888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69703746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚976700347401216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113302417 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32511699 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3524496620947960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12732310817792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12712336456696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219228090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113292339 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69704118 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37192414 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32511704ùú‘|-4¢3/ _wandbruntime‚22165  _runtime‚22165N´Ð £3‡ epoch‚43 2loss/SRqqt1/assignment_loss‚0.5075926227152718 /loss/FRt1/detection_loss‚0.4561090325984484 /loss/FRt2/detection_loss‚1.4514228377686438 1loss/SRqqt1/detection_loss‚0.8629421479636961 /loss/FBt2/detection_loss‚0.5303792274210155 %loss/total_loss‚7.739397995247372 /loss/FRt1/assignment_loss‚1.619145873542462 0loss/FRt2/assignment_loss‚0.6578328746436822 3loss/FBt1/assignment_loss‚9.297782795279036e-06 trainer/global_step‚12199 1loss/SRqqt2/detection_loss‚0.6193019153214232 /loss/FBt1/detection_loss‚0.5477834151991244 _step‚1018 2loss/SRqqt2/assignment_loss‚0.4814855252451843 2loss/FBt2/assignment_loss‚0.005393225045624962 % _timestamp‚1.7848614023604326e+09 _runtime‚22165.717623917„çm¤3Œ 0loss/FRt2/assignment_loss‚0.6578328746436822 3loss/FBt1/assignment_loss‚9.297782795279036e-06 trainer/global_step‚12199 1loss/SRqqt2/detection_loss‚0.6193019153214232 /loss/FBt1/detection_loss‚0.5477834151991244 _step‚1018 2loss/SRqqt2/assignment_loss‚0.4814855252451843 2loss/FBt2/assignment_loss‚0.005393225045624962 % _timestamp‚1.7848614023604326e+09 _runtime‚22165.717623917 epoch‚43 2loss/SRqqt1/assignment_loss‚0.5075926227152718 /loss/FRt1/detection_loss‚0.4561090325984484 /loss/FRt2/detection_loss‚1.4514228377686438 1loss/SRqqt1/detection_loss‚0.8629421479636961 /loss/FBt2/detection_loss‚0.5303792274210155 %loss/total_loss‚7.739397995247372 /loss/FRt1/assignment_loss‚1.619145873542462ú‘°!E¥3‚(:º  Þ¥‹ÓÉðŠ´) proc.memory.availableMB‚ 1926.57421875 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625! disk.dm-0.out‚53107.470703125 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.sda2.out‚ 880.1484375 disk.drbd1289.out‚0) disk./.usagePercent‚70.51462821733945 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0$ disk.nvme0n1.out‚53107.470703125 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1285.in‚0 cpu‚25.339993964981 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 110.60546875 disk.drbd1550.in‚0 disk.sda.out‚ 880.1484375 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0" proc.memory.rssMB‚ 2948.3359375 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0# memory_percent‚92.1607494354248 proc.cpu.threads‚17 network.sent‚ 2.1240376e+07 disk.drbd2041.out‚0* proc.memory.percent‚11.996809641520183 disk.drbd1584.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1356.in‚0 disk.drbd1817.out‚0 disk.drbd1356.out‚0% disk./.usageGB‚1260.8731689453125 disk.drbd1510.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 network.recv‚ 1.586156e+06¦¬6œ¦3‚(:‘ Þ¥‹ÓŒ¾ë³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚295.78' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.59333333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GøÜÐJ§3‚(:¿  í¥‹ÓþĆ´ disk.rbd1.out‚ 110.84765625 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0# memory_percent‚92.1607494354248) proc.memory.availableMB‚ 1926.57421875 disk.sda1.in‚0 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 119.03125" proc.memory.rssMB‚ 2948.3359375 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 cpu‚25.339933286254205 disk.drbd2118.out‚0 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.8731498718262 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 880.55859375! disk.dm-0.out‚53107.650390625 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 network.sent‚ 2.1240484e+07 network.recv‚ 1.586264e+06 disk.sda2.out‚ 880.55859375 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.996809641520183 disk.drbd1817.out‚0 disk.drbd1673.in‚0$ disk.nvme0n1.out‚53107.650390625) disk./.usagePercent‚70.51462715065024 disk.drbd1673.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 119.03125w‘Þ*¨3‚(:’ í¥‹ÓÀþâ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚182.225' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.74166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿›¨IK©3‚(:À  ü¥‹Óõ¨ö³ disk.drbd2078.in‚0 disk.drbd1356.in‚0% disk.nvme0n1.out‚53108.2294921875 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 1926.57421875 disk.drbd1938.in‚0 disk.drbd1817.in‚0 network.sent‚ 2.1242977e+07 network.recv‚ 1.587079e+06) disk./.usagePercent‚70.51462715065024 disk.drbd1673.in‚0" disk.dm-0.out‚53108.2294921875 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.in‚0 disk.drbd1828.in‚0 cpu‚25.33987291381124 disk.sda2.out‚ 880.94921875 disk.drbd1356.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0# memory_percent‚92.1607494354248 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd2118.out‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd2041.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 880.94921875 disk.rbd1.out‚ 111.02734375 disk.drbd1550.out‚0 disk.drbd1965.out‚0" proc.memory.rssMB‚ 2948.3359375* proc.memory.percent‚11.996809641520183% disk./.usageGB‚1260.8731498718262 disk.sda1.out‚0 disk.drbd1289.in‚0s'îª3‚(:’ ü¥‹ÓÐâ׳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.673' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.22433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÅXò’«3‚(:‡ ‹¦‹ÓУæ³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚66.0 gpu.0.powerWatts‚291.897' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.299 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™!ö]K¬3‚(:À  ‹¦‹Ó’ʆ´ disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0" proc.memory.rssMB‚ 2948.3359375 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0 network.sent‚ 2.1243043e+07 disk.sda.in‚ 0.03515625 disk.drbd1673.out‚0 disk.sda2.in‚ 0.03515625* proc.memory.percent‚11.996809641520183 network.recv‚ 1.587145e+06 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8722343444824 disk.dm-0.in‚ 119.03125 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1926.82421875" disk.dm-0.out‚53108.5615234375 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 119.03125$ memory_percent‚92.15973218282065 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1510.out‚0 disk.drbd1552.in‚0 cpu‚25.339845729056282 disk.drbd1584.in‚0) disk./.usagePercent‚70.51457594956844 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.sda2.out‚ 881.2890625 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 881.2890625 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 111.19140625% disk.nvme0n1.out‚53108.5615234375 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1931.out‚0Ó¢ç­3‚(:’ š¦‹Ó·¿Ñ³ gpu.0.gpu‚92.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚109.498' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.49933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏNz‰A®3‚(:¶  š¦‹ÓàÛò³ disk.drbd1510.in‚0$ disk./.usageGB‚1260.872215270996 disk.rbd1.out‚ 111.19140625" proc.memory.rssMB‚ 2948.3359375* proc.memory.percent‚11.996809641520183 network.recv‚ 1.587253e+06 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚881.625 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 119.03125% disk.nvme0n1.out‚53108.9521484375 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.out‚881.625 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 cpu‚25.339796797702604 disk.drbd2118.out‚0 disk.drbd1938.out‚0" disk.dm-0.out‚53108.9521484375 disk.drbd1828.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1931.in‚0) disk./.usagePercent‚70.51457488287923 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.15536117553711 network.sent‚ 2.1243151e+07 disk.drbd1552.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0( proc.memory.availableMB‚ 1927.8984375 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1552.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0¿yãJ¯3‚(:¿  ©¦‹Óîšø³" proc.memory.rssMB‚ 2948.3359375 disk.drbd2118.in‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 882.08984375 disk.drbd1938.out‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.996809641520183) disk./.usagePercent‚70.51457488287923$ disk./.usageGB‚1260.872215270996 disk.drbd1931.out‚0" disk.dm-0.out‚53109.1279296875 disk.drbd1673.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.15536117553711 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.sda.in‚ 0.03515625 disk.drbd1965.in‚0 disk.rbd1.out‚ 111.25390625% disk.nvme0n1.out‚53109.1279296875 disk.drbd1226.in‚0 disk.drbd1285.out‚0 network.sent‚ 2.1249256e+07 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.drbd1817.out‚0 network.recv‚ 1.588201e+06 disk.drbd2118.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 119.03125 disk.sda.out‚ 882.08984375 disk.dm-0.in‚ 119.03125 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1927.8984375 disk.drbd1584.in‚0 cpu‚25.33974745981207 disk.drbd2041.in‚0 disk.drbd1828.in‚0ÒyJÜ°3‚(:’ ©¦‹ÓϬܳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.563' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.85433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°;ØÎ4±3/ _wandbruntime‚22251  _runtime‚22251îaÌ0Ȳ3 % lr-AdamW/pg1‚4.64453908809678e-06 % lr-AdamW/pg2‚4.64453908809678e-06 trainer/global_step‚12249 % _timestamp‚1.7848614876791337e+09 _runtime‚22251.036391781 _step‚1019äôv«Í³3Ç % _timestamp‚1.7848614876791337e+09 _runtime‚22251.036391781 _step‚1019 % lr-AdamW/pg1‚4.64453908809678e-06 % lr-AdamW/pg2‚4.64453908809678e-06 trainer/global_step‚12249ûí ÿš4´3/ _wandbruntime‚22251  _runtime‚22251=‘(µ3“P GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37338217 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12762419537752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12782477865984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32641064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚964981525192704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69979276 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12783823302144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3539619815179776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12782477865984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12761074926466 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3526835965704192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113749273 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3526835965704192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 % _timestamp‚1.7848614876802764e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚980612159298048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚980611737842688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15630580741120 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106345734 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3526835991877632 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113739209 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37338601 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚22251.04094203 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3539618443570176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220084943 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3539618443570176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3526835965704192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113749273 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3526835965704192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 % _timestamp‚1.7848614876802764e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚980612159298048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚980611737842688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15630580741120 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106345734 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3526835991877632 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113739209 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37338601 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚22251.04094203 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3539618443570176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220084943 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3539618443570176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12784845817344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113748291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3527114874994688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37341562 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32643699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚980691557650432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220112476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220112476 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3527114901168128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 trainer/global_step‚12249 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3539898375368704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113758369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37341175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3525724373180064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784861489156016e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12783500374016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69984874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220102392 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _step‚1021žŠYh('¹3¢N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220112476 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3527114901168128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 trainer/global_step‚12249 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3539898375368704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113758369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37341175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3525724373180064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784861489156016e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12783500374016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69984874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220102392 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 _step‚1021 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106354107 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69985266 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3525724347752096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12784845817344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113748291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3527114874994688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37341562 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32643699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚980691557650432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220112476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224ýÀy¿N4º3/ _wandbruntime‚22252  _runtime‚22252ÅÂì¼ »3† epoch‚43 % _timestamp‚1.7848614891942284e+09 _runtime‚22252.551454111 0loss/FRt1/assignment_loss‚1.7145527274641272 %loss/total_loss‚7.839052401739144 _step‚1022 /loss/FRt2/detection_loss‚1.4291182453940872 1loss/SRqqt2/detection_loss‚0.6500256453809579 0loss/FRt2/assignment_loss‚0.7800625266043792 2loss/SRqqt1/assignment_loss‚0.5722018995564072 1loss/FBt1/assignment_loss‚0.04986600754567035 2loss/FBt2/assignment_loss‚0.020570296164830568 1loss/SRqqt1/detection_loss‚0.8372072461472407 /loss/FBt1/detection_loss‚0.4049730221449863 trainer/global_step‚12249 2loss/SRqqt2/assignment_loss‚0.4851045172083739 /loss/FRt1/detection_loss‚0.5303624506179275 /loss/FBt2/detection_loss‚0.3650078175101549M¸¼3‹ 1loss/FBt1/assignment_loss‚0.04986600754567035 2loss/FBt2/assignment_loss‚0.020570296164830568 1loss/SRqqt1/detection_loss‚0.8372072461472407 /loss/FBt1/detection_loss‚0.4049730221449863 trainer/global_step‚12249 2loss/SRqqt2/assignment_loss‚0.4851045172083739 /loss/FRt1/detection_loss‚0.5303624506179275 /loss/FBt2/detection_loss‚0.3650078175101549 epoch‚43 % _timestamp‚1.7848614891942284e+09 _runtime‚22252.551454111 0loss/FRt1/assignment_loss‚1.7145527274641272 %loss/total_loss‚7.839052401739144 _step‚1022 /loss/FRt2/detection_loss‚1.4291182453940872 1loss/SRqqt2/detection_loss‚0.6500256453809579 0loss/FRt2/assignment_loss‚0.7800625266043792 2loss/SRqqt1/assignment_loss‚0.5722018995564072þI©ŠÒ’½3‚(:‡ ¸¦‹Ó۾г gpu.0.gpu‚83.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.623' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.541 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m÷Ýö`¾3‚(:¿  ¸¦‹Ó—ºí³ disk.drbd1356.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.15548833211264* proc.memory.percent‚11.996809641520183 network.sent‚ 2.1249364e+07 disk.drbd1550.in‚0 disk.sda2.out‚ 882.40234375 proc.cpu.threads‚17 disk.drbd1226.out‚0$ disk./.usageGB‚1260.872215270996 disk.drbd1552.ou‰3Ùêt‚0 disk.sda2.in‚ 0.03515625 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 882.40234375 disk.sda.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 111.25390625( proc.memory.availableMB‚ 1927.8671875 disk.drbd2041.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51457488287923 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚53109.4560546875 disk.drbd1931.in‚0 disk.drbd1938.in‚0 cpu‚25.33973212706392 disk.dm-0.in‚ 119.03125 network.recv‚ 1.588309e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0" proc.memory.rssMB‚ 2948.3359375 disk.drbd1550.out‚0 disk.drbd1584.in‚0" disk.dm-0.out‚53109.45605468750^ã¿3‚(:’ Ǧ‹ÓÍ£Û³ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.861' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.62033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ§óHÀ3‚(:½  Ǧ‹Óþ˜û³ disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.rbd1.out‚111.4736328125) proc.memory.availableMB‚ 1928.08203125 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51457488287923 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 cpu‚25.33970515587381 disk.drbd1938.out‚0$ disk./.usageGB‚1260.872215270996 disk.sda.out‚ 882.671875 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-0.in‚ 119.03125% disk.nvme0n1.out‚53109.6552734375 disk.drbd1965.out‚0$ memory_percent‚92.15461413065592 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1965.in‚0 disk.drbd1828.out‚0" proc.memory.rssMB‚ 2948.3359375* proc.memory.percent‚11.996809641520183 network.sent‚ 2.124943e+07 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0" disk.dm-0.out‚53109.6552734375 disk.drbd2041.in‚0 network.recv‚ 1.588375e+06 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 882.671875 disk.drbd2078.in‚0 disk.drbd1356.in‚0_ó„*Á3‚(:’ Ö¦‹Ó •Ý³ gpu.0.gpu‚91.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚295.408' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.46933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o’&PÂ3‚(:Å  Ö¦‹ÓÔüú³ disk.drbd1226.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.15561548868814 cpu‚25.339656081731096 disk.drbd1965.out‚0& disk.nvme0n1.out‚53110.31884765625 disk.drbd1673.in‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51457530955491 disk.drbd1828.out‚0 disk.dm-0.in‚ 119.03125% disk./.usageGB‚1260.8722229003906 disk.drbd1552.in‚0 disk.sda.out‚ 882.97265625 disk.drbd1673.out‚0 network.sent‚ 2.1249538e+07 disk.drbd1285.out‚0 disk.drbd1817.in‚0# disk.dm-0.out‚53110.31884765625 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd2041.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 882.97265625 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚111.4736328125( proc.memory.availableMB‚ 1927.8359375 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0" proc.memory.rssMB‚ 2948.3359375 network.recv‚ 1.588483e+06 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.996809641520183 disk.drbd1226.in‚0 disk.sda.in‚ 0.03515625 disk.nvme0n1.in‚ 119.03125É5™×“Ã3‚(:ˆ 妋Ó·Üó gpu.0.gpu‚93.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚329.154' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚109.718 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô·Ä8KÄ3‚(:À  妋ÓáŽâ³* proc.memory.percent‚11.996809641520183 disk.drbd1938.in‚0) proc.memory.availableMB‚ 1927.83203125 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1584.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2948.3359375$ disk./.usageGB‚1260.872329711914 disk.drbd2041.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1550.in‚0 disk.drbd1285.in‚0# disk.dm-0.out‚53110.50634765625 disk.drbd1510.out‚0) disk./.usagePercent‚70.51458128301445 disk.drbd1356.out‚0& disk.nvme0n1.out‚53110.50634765625 cpu‚25.339618724369856 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1938.out‚0 network.recv‚ 1.589281e+06 disk.drbd2111.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 883.265625# memory_percent‚92.1556313832601 disk.drbd1965.out‚0 disk.drbd1931.in‚0 disk.drbd1226.in‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1817.out‚0 network.sent‚ 2.1255639e+07 disk.drbd1828.in‚0 disk.rbd1.out‚111.5361328125 disk.drbd1673.out‚0 disk.sda2.out‚ 883.265625 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0ÏÓ¶ãÅ3‚(:’ ô¦‹Ó×À‘µ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚280.964' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.65466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý²îÿOÆ3‚(:Ä  ô¦‹ÓÁí­µ disk.drbd1931.out‚0# disk.dm-0.out‚53110.87060546875 disk.drbd2078.in‚0) proc.memory.availableMB‚ 1927.83203125 disk.drbd1938.out‚0% disk./.usageGB‚1260.8723106384277 disk.rbd1.out‚111.5361328125 disk.drbd1289.out‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0# memory_percent‚92.1556313832601 proc.cpu.threads‚17 disk.drbd1285.in‚0& disk.nvme0n1.out‚53110.87060546875* proc.memory.percent‚11.996809641520183 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.out‚ 883.62890625 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0 disk.sda.out‚ 883.62890625 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0) disk./.usagePercent‚70.51458021632526 disk.drbd1673.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1828.out‚0 cpu‚25.33955480317554 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.drbd1356.in‚0 network.sent‚ 2.1255705e+07 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2948.3359375 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.dm-0.in‚ 119.03125 network.recv‚ 1.589347e+06 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1356.out‚0EcbžÇ3‚(:“ ƒ§‹ÓšÀ­³ gpu.0.gpu‚98.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚307.141' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.38033333333334 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íû›òNÈ3‚(:à  ƒ§‹ÓÛ¨Ô³ disk.dm-0.in‚ 119.03125# disk.dm-0.out‚53111.30810546875 disk.drbd1356.out‚0 disk.drbd2111.in‚0$ memory_percent‚92.15561548868814 network.sent‚ 2.1255771e+07 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0( proc.memory.availableMB‚ 1927.8359375 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 network.recv‚ 1.589413e+06 disk.drbd1552.out‚0 disk.sda.out‚ 884.02734375 disk.drbd1938.in‚0* proc.memory.percent‚11.996809641520183 cpu‚25.33950995858375 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚53111.30810546875 disk.sda1.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0" proc.memory.rssMB‚ 2948.3359375 disk.nvme0n1.in‚ 119.03125 disk.drbd1817.out‚0 disk.sda.in‚ 0.03515625 disk.sda2.in‚ 0.03515625 disk.drbd1289.out‚0) disk./.usagePercent‚70.51458021632526 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.rbd1.out‚ 111.669921875 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 884.02734375 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8723106384277 disk.drbd1510.in‚0 disk.drbd1356.in‚0š`á4É3/ _wandbruntime‚22337  _runtime‚22337;RÇÊ3Á $ _timestamp‚1.784861574571101e+09 _runtime‚22337.928347457 _step‚1023 % lr-AdamW/pg1‚4.38639089527304e-06 % lr-AdamW/pg2‚4.38639089527304e-06 trainer/global_step‚12299&ÙÚwÌË3Æ _step‚1023 % lr-AdamW/pg1‚4.38639089527304e-06 % lr-AdamW/pg2‚4.38639089527304e-06 trainer/global_step‚12299 $ _timestamp‚1.784861574571101e+09 _runtime‚22337.928347457ÿÇQ5µ4Ì3/ _wandbruntime‚22337  _runtime‚22337N½(Í3“P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12813523420560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3552199183515648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 _step‚1024 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106764090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114205225 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3539385634667120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220969315 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114205225 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32773064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 trainer/global_step‚12299 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12812178809274 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106764084 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3540781456402432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚968910475399168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12835012858368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚984603663447552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3540781456402432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37487524 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚968910843490304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15692766593024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3540781430228992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70260195 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3553615097651200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12833667422208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 _runtime‚22337.932884354 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3553616469260800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32773059 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114195161 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3540781430228992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚388 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15692819957248 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3552197813476394 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12833667422208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3553615097651200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12833667422208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 _runtime‚22337.932884354 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3553616469260800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32773059 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114195161 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3540781430228992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚388 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15692819957248 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3552197813476394 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12833667422208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12835012858368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚984603663447552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3540781456402432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37487524 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚968910843490304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15692766593024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0€„bp4Ï3/ _wandbruntime‚22339  _runtime‚22339)t^¾£Ð3žN JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitu ïr&or.on_train_batch_end/requested_bytes.small_pool.freed‚12813199610914 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114204243 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70265809 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848615760389361e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚984683491200512 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚968989061765120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32775704 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3552477636051810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15694005890048 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3539664436440896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12836035373568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3541060339519488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15694061344256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12834689930240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 _runtime‚22339.396513759 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3541060365692928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220986778 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚1025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220976694 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106772451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106772457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12814544222312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚12299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220976694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚389 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3553896401066496 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3541060339519488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3552479006091176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3539664461868864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114214321 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚968989429856256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12836035373568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114204243 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114214321 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3541060365692928 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3553896401066496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32775699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12834689930240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37490110 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825ÿ,B_)'Ñ3£N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚968989061765120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32775704 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3552477636051810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15694005890048 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3539664436440896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12836035373568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3541060339519488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15694061344256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12834689930240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 _runtime‚22339.396513759 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3541060365692928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220986778 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚1025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220976694 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106772451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106772457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12814544222312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 trainer/global_step‚12299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220976694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚389 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3553896401066496 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3541060339519488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3552479006091176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3539664461868864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114214321 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚968989429856256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12836035373568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114204243 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114214321 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3541060365692928 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3553896401066496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32775699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12834689930240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37490110 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12813199610914 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114204243 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70265809 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7848615760389361e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚984683491200512 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366ÿÑá¢4Ò3/ _wandbruntime‚22339  _runtime‚22339DK[MÓ3‰ 3loss/SRqqt2/assignment_loss‚0.46740820913083464 4loss/FBt1/assignment_loss‚0.00035945732214210665 /loss/FRt2/detection_loss‚1.3915666194050573 1loss/SRqqt1/detection_loss‚0.7896673949439182 trainer/global_step‚12299 0loss/FRt1/assignment_loss‚1.5514496210605417 1loss/FBt2/assignment_loss‚0.03186646811671907 /loss/FRt1/detection_loss‚0.5156293840183043 %loss/total_loss‚7.487874559420897 % _timestamp‚1.7848615760824027e+09 _step‚1026 0loss/FRt2/assignment_loss‚0.6981131533330137 /loss/FBt2/detection_loss‚0.4104839121219107 _runtime‚22339.439573528 2loss/SRqqt1/assignment_loss‚0.4140556265528823 1loss/SRqqt2/detection_loss‚0.6172202217373762 /loss/FBt1/detection_loss‚0.6000544916781964 epoch‚43ÔÌÐÔ3Ž _runtime‚22339.439573528 2loss/SRqqt1/assignment_loss‚0.4140556265528823 1loss/SRqqt2/detection_loss‚0.6172202217373762 /loss/FBt1/detection_loss‚0.6000544916781964 epoch‚43 3loss/SRqqt2/assignment_loss‚0.46740820913083464 4loss/FBt1/assignment_loss‚0.00035945732214210665 /loss/FRt2/detection_loss‚1.3915666194050573 1loss/SRqqt1/detection_loss‚0.7896673949439182 trainer/global_step‚12299 0loss/FRt1/assignment_loss‚1.5514496210605417 1loss/FBt2/assignment_loss‚0.03186646811671907 /loss/FRt1/detection_loss‚0.5156293840183043 %loss/total_loss‚7.487874559420897 % _timestamp‚1.7848615760824027e+09 _step‚1026 0loss/FRt2/assignment_loss‚0.6981131533330137 /loss/FBt2/detection_loss‚0.4104839121219107‚è}°‘Õ3‚(:† ’§‹Ó®Ó®³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚308.001' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.667 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Ö§IÖ3‚(:¾  ’§‹ÓÑÕγ disk.drbd1584.in‚0 disk.sda.out‚ 884.3984375 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0" proc.memory.rssMB‚ 2948.3359375 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 network.recv‚ 1.594645e+06 disk.sda.in‚ 0.03515625 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1552.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚53111.77978515625 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.sda1.in‚0* proc.memory.percent‚11.996809641520183 disk.drbd2118.in‚0# disk.dm-0.out‚53111.77978515625 disk.rbd1.out‚ 111.669921875 disk.drbd1356.in‚0 disk.drbd1550.out‚0 cpu‚25.339461251882142 disk.drbd2118.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1828.out‚0 disk.drbd1285.out‚0$ proc.memory.availableMB‚1927.625 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0$ memory_percent‚92.15647379557291 disk.sda2.out‚ 884.3984375) disk./.usagePercent‚70.51447376074267 disk.drbd1673.out‚0 disk.drbd1510.out‚0 network.sent‚ 2.1428657e+07 disk.drbd1584.out‚0 disk.sda2.in‚ 0.03515625 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1550.in‚0% disk./.usageGB‚1260.8704071044922Q¤˜}×3‚(:’ ¡§‹ÓŠ‡Ó³ gpu.0.gpu‚71.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚128.32' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚42.773333333333326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þN|LØ3‚(:Á  ¡§‹Ó÷üñ³ disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0# disk.dm-0.out‚53111.97119140625 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51447376074267 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0$ memory_percent‚92.15714136759439 cpu‚25.33938952538897 network.recv‚ 1.594777e+06 disk.sda2.in‚ 0.03515625 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 884.6328125 disk.drbd1673.in‚0 disk.sda2.out‚ 884.6328125 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd1552.in‚0" proc.memory.rssMB‚ 2948.3359375 network.sent‚ 2.1429451e+07 disk.drbd2078.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.996809641520183 disk.drbd1289.in‚0 disk.dm-0.in‚ 119.03125 disk.nvme0n1.in‚ 119.03125 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 111.732421875 disk.drbd1552.out‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚53111.97119140625 disk.drbd1226.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1927.4609375% disk./.usageGB‚1260.8704071044922 disk.sda.in‚ 0.03515625 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1965.out‚0 disk.drbd1550.out‚0íÃHÙ3‚(:’ °§‹Óöêϵ gpu.0.gpu‚99.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚294.596' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.19866666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B,ûNÚ3‚(:à  °§‹ÓõÍèµ disk.drbd2111.out‚0( proc.memory.availableMB‚ 1925.9609375# proc.memory.rssMB‚ 2949.65234375 disk.sda.in‚ 0.03515625" disk.dm-0.out‚53112.3525390625 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 cpu‚25.340016679189077 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 119.03125$ memory_percent‚92.16324488321939 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 119.03125 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚53112.3525390625 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 network.sent‚ 2.1434707e+07 disk.drbd1931.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1356.out‚0 disk.drbd1510.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8628273010254 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51404985845289 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 network.recv‚ 1.596811e+06 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.rbd1.out‚ 111.732421875* proc.memory.percent‚12.002166112263996 disk.sda.out‚ 884.99609375 disk.sda2.out‚ 884.99609375 disk.drbd2041.out‚0¦ô_+4Û3/ _wandbruntime‚22392  _runtime‚223926,<†ÂÜ3v ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35840871930122375 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4549259841442108 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03302428871393204 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.920347273349762 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.703504741191864 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030784036964178085 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.713350236415863 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9832039475440979 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3869612216949463 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 _runtime‚22392.56808508 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6088539958000183 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2993394732475281 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7288841605186462 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7121766805648804 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31064555048942566 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8126803636550903 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28098806738853455 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3495486080646515 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7171660661697388 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8519379496574402 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435050368309021 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38990435004234314 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.663813591003418 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27208787202835083 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8032118678092957 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2924017310142517 +jet/accuracy_3_of_3‚0.07724948227405548 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27535951137542725 6validation_average_jet_accuracy‚0.4090573191642761 _step‚1027 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28511184453964233 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.780295193195343 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5415826439857483 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45588067173957825 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8103574514389038 8"Purity/2FRt0SRqqt*FBt/event_purity‚0.176265686750412 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6983309388160706 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6513155102729797 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7848616292100277e+09 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3086486756801605 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6471889615058899 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14447593688964844 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 &particle/accuracy‚0.85551518201828 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29636600613594055 7 Purity/*FRt*SRqqt2·<ÑÍQ'FBt/FBt_purity‚0.9971320033073425 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746345639228821 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 +jet/accuracy_3_of_4‚0.11347367614507675 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1746649444103241 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 7"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.77507084608078 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490260243415833 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35840871930122375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38935309648513794 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1866229772567749 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3612113893032074 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38953515887260437 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8099468946456909 *jet/accuracy_1_of_1‚0.3754619061946869 trainer/global_step‚12319 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6591455340385437 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3694395422935486 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7029848694801331 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.883858859539032 +particle/sensitivity‚0.6417928338050842 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35875365138053894 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7273203730583191 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37201371788978577 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 epoch‚43 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7521430850028992 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35756561160087585 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6822500228881836 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6513155102729797 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8090339303016663 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 +particle/specificity‚0.9138907194137573 'particle/f_score‚0.6558612585067749 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2813911736011505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6721619367599487 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2228589802980423 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45674628019332886 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026249166578054428 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 *jet/accuracy_2_of_3‚0.6067718863487244 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027860669419169426 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22635693848133087 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5988618731498718 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.044147610664367676 *jet/accuracy_1_of_2‚0.7945643663406372 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6277090311050415 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08880007266998291 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.02295798622071743 /particle/accuracy_3_of_3‚0.9993678331375122 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.156746506690979 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6454356908798218 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16878582537174225 particle/accuracy_1_of_4‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4073679745197296 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_2‚1 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0718776062130928 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6949257254600525 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6266483068466187 )loss/val_total_loss‚7.724052140194132 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6742180585861206 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6634030342102051 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.27171605825424194 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9495930671691895 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9671996831893921 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.17218835651874542 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5831097960472107 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 *jet/accuracy_1_of_3‚0.9422892332077026 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.627723753452301 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7317937612533569 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7669938802719116 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6387776136398315 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5114666819572449 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8785568475723267 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5216056108474731 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8352271914482117 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7260070443153381 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05316916108131409 *jet/accuracy_2_of_4‚0.8421576619148254 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7456907629966736 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8541358113288879 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.37027284502983093 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1yn;Ý3’v 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8032118678092957 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2924017310142517 +jet/accuracy_3_of_3‚0.07724948227405548 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9821428656578064 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27535951137542725 6validation_average_jet_accuracy‚0.4090573191642761 _step‚1027 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28511184453964233 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.780295193195343 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5415826439857483 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45588067173957825 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8103574514389038 8"Purity/2FRt0SRqqt*FBt/event_purity‚0.176265686750412 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6983309388160706 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6513155102729797 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7848616292100277e+09 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3086486756801605 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6471889615058899 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14447593688964844 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 &particle/accuracy‚0.85551518201828 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29636600613594055 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746345639228821 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 +jet/accuracy_3_of_4‚0.11347367614507675 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1746649444103241 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 7"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.77507084608078 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490260243415833 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35840871930122375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38935309648513794 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1866229772567749 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3612113893032074 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38953515887260437 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8099468946456909 *jet/accuracy_1_of_1‚0.3754619061946869 trainer/global_step‚12319 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6591455340385437 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3694395422935486 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7029848694801331 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.883858859539032 +particle/sensitivity‚0.6417928338050842 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35875365138053894 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7273203730583191 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37201371788978577 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.850649356842041 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 epoch‚43 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7521430850028992 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35756561160087585 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6822500228881836 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6513155102729797 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8090339303016663 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 +particle/specificity‚0.9138907194137573 'particle/f_score‚0.6558612585067749 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2813911736011505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6721619367599487 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2228589802980423 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45674628019332886 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026249166578054428 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 *jet/accuracy_2_of_3‚0.6067718863487244 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027860669419169426 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22635693848133087 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5988618731498718 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.044147610664367676 *jet/accuracy_1_of_2‚0.7945643663406372 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6277090311050415 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08880007266998291 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.02295798622071743 /particle/accuracy_3_of_3‚0.9993678331375122 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.156746506690979 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6454356908798218 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16878582537174225 particle/accuracy_1_of_4‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4073679745197296 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.9642857313156128 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 particle/accuracy_1_of_2‚1 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0718776062130928 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6949257254600525 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6266483068466187 )loss/val_total_loss‚7.724052140194132 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6742180585861206 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6634030342102051 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.27171605825424194 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9495930671691895 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9671996831893921 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.17218835651874542 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5831097960472107 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 *jet/accuracy_1_of_3‚0.9422892332077026 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.627723753452301 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7317937612533569 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7669938802719116 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6387776136398315 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5114666819572449 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8785568475723267 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5216056108474731 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8352271914482117 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7260070443153381 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05316916108131409 *jet/accuracy_2_of_4‚0.8421576619148254 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7456907629966736 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8541358113288879 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.37027284502983093 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35840871930122375 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4549259841442108 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03302428871393204 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.920347273349762 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.703504741191864 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030784036964178085 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.713350236415863 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9832039475440979 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3869612216949463 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 _runtime‚22392.56808508 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6088539958000183 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2993394732475281 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7288841605186462 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9821428656578064 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7121766805648804 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31064555048942566 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8126803636550903 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28098806738853455 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3495486080646515 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7171660661697388 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8519379496574402 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435050368309021 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38990435004234314 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9642857313156128 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.663813591003418 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27208787202835083 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675ƒ6‰*›Þ3‚(: ¿§‹Ó€Øͳ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚58.0 gpu.0.powerWatts‚106.11' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.370000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘léJFß3‚(:»  ¿§‹ÓÖ˜ë³ disk.drbd1552.out‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚53112.8564453125 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1584.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.rbd1.out‚111.8662109375& proc.memory.availableMB‚ 4.50390625 disk.drbd2118.out‚0" disk.dm-0.out‚53112.8564453125 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 network.sent‚ 2.143568e+07 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 network.recv‚ 1.597729e+06 disk.sda.in‚ 0.03515625 disk.drbd1828.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1828.out‚0 disk.drbd1510.out‚0$ memory_percent‚99.98167355855307 proc.memory.rssMB‚ 3919.1875 disk.drbd2118.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚15.947214762369791 cpu‚25.341432503356604 disk.sda.out‚ 885.3359375 disk.drbd1356.out‚0) disk./.usagePercent‚70.62358305272865 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1356.in‚0% disk./.usageGB‚1262.8213920593262 disk.sda2.out‚ 885.3359375 disk.drbd2111.out‚0-ifà3‚(:’ Χ‹ÓäÐͳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚294.563' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.18766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ÉRLá3‚(:Á  Χ‹Ó¢ó³ network.recv‚ 1.599458e+06 disk.drbd2078.in‚0 cpu‚25.340780879938233 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚111.8662109375 disk.drbd1550.out‚0$ memory_percent‚88.13125292460123# proc.memory.rssMB‚ 2949.66015625 disk.drbd1673.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1226.out‚0 disk.drbd1965.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚54082.75048828125 disk.drbd1510.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2916.86328125) disk./.usagePercent‚70.51448698768881 disk.drbd1356.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 885.65625 network.sent‚ 2.1491612e+07 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda.in‚ 0.03515625 disk.sda.out‚ 885.65625 disk.drbd2111.out‚0% disk./.usageGB‚1260.8706436157227 disk.drbd1817.in‚0 disk.drbd1673.in‚0* proc.memory.percent‚12.002197901407879 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1584.out‚0# disk.dm-0.out‚54082.75048828125ì5©$â3‚(:’ ݧ‹ÓŒíã³ gpu.0.gpu‚97.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.343' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.44766666666666 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o™_åQã3‚(:Æ  ݧ‹ÓŒßˆ´ disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2041.in‚0 disk.rbd1.out‚111.9287109375# proc.memory.rssMB‚ 2949.66015625 disk.drbd1817.in‚0 disk.sda2.in‚ 0.03515625 disk.drbd1226.in‚0 cpu‚25.340742122632907 disk.drbd1817.out‚0 disk.drbd1965.in‚0 network.sent‚ 2.1494611e+07 disk.drbd1931.out‚0 disk.drbd1938.in‚0) disk./.usagePercent‚70.51426831640192 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 886.08984375* proc.memory.percent‚12.002197901407879 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 886.08984375( proc.memory.availableMB‚ 3852.2578125 disk.drbd2111.in‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚55052.55126953125 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.out‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚55052.55126953125 disk.drbd1673.in‚0$ memory_percent‚84.32512283325195 disk.drbd1584.out‚0% disk./.usageGB‚1260.8667335510254 disk.drbd1828.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 network.recv‚ 1.600459e+06 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.sda.in‚ 0.03515625jsÐ žä3‚(:“ 짋Óšþij gpu.0.gpu‚61.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚300.902' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.30066666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^“³ûå3‚(:  짋Ó“Úê³) disk./.usagePercent‚70.51425380942875 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 cpu‚25.340704357658627 disk.sda2.in‚ 0.0390625 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2949.66015625 disk.sda.out‚ 889.75390625 disk.drbd2078.out‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 3603.21875 network.sent‚ 2.1497615e+07 disk.drbd1673.in‚0 disk.drbd1965.out‚0* proc.memory.percent‚12.002197901407879 disk.drbd1931.in‚0$ memory_percent‚85.3Éwb†R3846537272134 disk.drbd2118.out‚0# disk.dm-0.out‚55053.16748046875 disk.drbd1828.out‚0& disk.nvme0n1.out‚55053.16748046875 disk.sda2.out‚ 889.75390625 disk.drbd1226.out‚0% disk./.usageGB‚1260.8664741516113 disk.drbd2041.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚111.9287109375 disk.sda.in‚ 0.0390625 disk.drbd1285.in‚0 disk.rbd1.in‚0 network.recv‚ 1.601492e+06 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.drbd1552.out‚0˜„W³4æ3/ _wandbruntime‚22448  _runtime‚22448Š´ÂÌç3Æ _step‚1028 ' lr-AdamW/pg1‚4.1353442127808965e-06 ' lr-AdamW/pg2‚4.1353442127808965e-06 trainer/global_step‚12349 % _timestamp‚1.7848616850646768e+09 _runtime‚22448.422845325yïSøÑè3Ë % _timestamp‚1.7848616850646768e+09 _runtime‚22448.422845325 _step‚1028 ' lr-AdamW/pg1‚4.1353442127808965e-06 ' lr-AdamW/pg2‚4.1353442127808965e-06 trainer/global_step‚12349„×>O4é3/ _wandbruntime‚22448  _runtime‚22448¢ö(ê3”P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3570178046896128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3570178046896128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3557287597651968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32911956 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107271872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70575856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37663900 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3568754986077738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 % _timestamp‚1.7848616850655246e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3555884724507088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚988651555381248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚988651138120192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3570179418505728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12890475417600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚323 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70575528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12868891531396 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12890475417600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚972889161300992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15762394080256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107271866 trainer/global_step‚12349 _step‚1029 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114708489 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 _runtime‚22448.423084879 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3555884749935056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114698425 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚972888793209856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚04(ë3™P ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12868891531396 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12890475417600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚972889161300992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15762394080256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107271866 trainer/global_step‚12349 _step‚1029 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114708489 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 _runtime‚22448.423084879 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3555884749935056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114698425 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚972888793209856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3570178046896128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3570178046896128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3557287597651968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32911956 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107271872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70575856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37663900 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3568754986077738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 % _timestamp‚1.7848616850655246e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3555884724507088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚988651555381248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚988651138120192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3570179418505728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12890475417600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚323 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70575528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0…*UÄó4ì3/ _wandbruntime‚22449  _runtime‚22449“2&$'í3žN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3557566480769024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12891497925632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107280239 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚988730981287424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3556163551708832 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3557566506942464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221997824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚972967747666944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107280233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12871256944434 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107280239 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114707507 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 % _timestamp‚1.7848616865438328e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3570459350311424 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3569033438613900 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221997824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12892843368960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221987740 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚1030 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107280233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚972967379575808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15763601711616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12892843368960 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70581431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37666515 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32914596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚325 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3570459350311424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114707507 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3557566506942464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12891497925632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3569034808653266 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3556163526280864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70581106 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15763648777216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚22449.901505609 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660w êsÒî3£N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107280233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚972967379575808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 ^KDeviceStatsMonitor.on_train_batch_enqsW#d/inactive_split_bytes.small_pool.freed‚15763601711616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12892843368960 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70581431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37666515 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32914596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚325 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3570459350311424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114707507 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3557566506942464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12891497925632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3569034808653266 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3556163526280864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70581106 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15763648777216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚22449.901505609 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3557566480769024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12891497925632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107280239 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚988730981287424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3556163551708832 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3557566506942464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221997824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚972967747666944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107280233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12871256944434 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107280239 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114707507 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 % _timestamp‚1.7848616865438328e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3570459350311424 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3569033438613900 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221997824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12892843368960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221987740 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚1030†ÞGöv4ï3/ _wandbruntime‚22449  _runtime‚22449—ðûIð3ˆ _step‚1031 2loss/SRqqt1/assignment_loss‚0.4693379063816652 epoch‚44 _runtime‚22449.93522907 1loss/SRqqt1/detection_loss‚0.8552551335614065 &loss/total_loss‚7.5839048103036095 4loss/FBt1/assignment_loss‚0.00021707017211743582 1loss/SRqqt2/detection_loss‚0.6104560080252073 /loss/FBt1/detection_loss‚0.6721436266632129 0loss/FBt2/detection_loss‚0.35807854875053424 trainer/global_step‚12349 $ _timestamp‚1.784861686577988e+09 0loss/FRt1/assignment_loss‚1.5567240552361963 0loss/FRt2/assignment_loss‚0.6598080310268678 1loss/SRqqt2/assignment_loss‚0.477437029627686 2loss/FBt2/assignment_loss‚0.016974746223847877 /loss/FRt1/detection_loss‚0.5244160331766378 /loss/FRt2/detection_loss‚1.3830566214582296÷`ÿ ñ3 1loss/SRqqt1/detection_loss‚0.8552551335614065 &loss/total_loss‚7.5839048103036095 4loss/FBt1/assignment_loss‚0.00021707017211743582 1loss/SRqqt2/detection_loss‚0.6104560080252073 /loss/FBt1/detection_loss‚0.6721436266632129 0loss/FBt2/detection_loss‚0.35807854875053424 trainer/global_step‚12349 $ _timestamp‚1.784861686577988e+09 0loss/FRt1/assignment_loss‚1.5567240552361963 0loss/FRt2/assignment_loss‚0.6598080310268678 1loss/SRqqt2/assignment_loss‚0.477437029627686 2loss/FBt2/assignment_loss‚0.016974746223847877 /loss/FRt1/detection_loss‚0.5244160331766378 /loss/FRt2/detection_loss‚1.3830566214582296 _step‚1031 2loss/SRqqt1/assignment_loss‚0.4693379063816652 epoch‚44 _runtime‚22449.93522907‡ÐŽƒNžò3‚(:“ û§‹ÓÜÜÖ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.932' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.97733333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jâ HFó3‚(:»  û§‹ÓæÑþ³ disk.drbd1289.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.0390625 disk.drbd1938.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 3314.34765625 network.sent‚ 2.1500442e+07 disk.drbd1828.in‚0" memory_percent‚86.513884862264 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1285.out‚0 disk.sda.out‚ 890.12109375 disk.sda2.in‚ 0.0390625& disk.nvme0n1.out‚55053.35498046875 disk.drbd2078.out‚0 network.recv‚ 1.60169e+06% disk./.usageGB‚1260.8664741516113 disk.rbd1.out‚112.0625 disk.sda2.out‚ 890.12109375 disk.drbd1828.out‚0 disk.drbd1817.in‚0 cpu‚25.34066571548006 disk.drbd2041.in‚0# disk.dm-0.out‚55053.35498046875 disk.rbd1.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2949.66015625* proc.memory.percent‚12.002197901407879 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51425380942875 disk.drbd1226.in‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0C}ÛÝô3‚(:’ Š¨‹Óüþ¹³ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚282.623' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.20766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙjS"Eõ3‚(:º  Š¨‹Ó«åÕ³ disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0( proc.memory.availableMB‚ 3005.3359375 disk.drbd1965.out‚0 disk.drbd2078.in‚0% disk.nvme0n1.out‚55054.0419921875 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 network.sent‚ 2.150055e+07) disk./.usagePercent‚70.51425274273954 disk.drbd2041.in‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 890.57421875 disk.drbd2041.out‚0 disk.drbd1817.out‚0" disk.dm-0.out‚55054.0419921875 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 network.recv‚ 1.601798e+06$ disk./.usageGB‚1260.866455078125 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2949.66015625 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.0390625 cpu‚25.340661582339152 disk.drbd1226.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2118.out‚0$ memory_percent‚87.77125676472983* proc.memory.percent‚12.002197901407879 disk.drbd2078.out‚0 disk.rbd1.out‚112.0625 disk.drbd1510.out‚0 disk.sda.out‚ 890.57421875 disk.sda2.in‚ 0.0390625 disk.drbd1931.out‚0 disk.sda1.out‚0«­iœö3‚(:‘ ™¨‹Óż±³ gpu.0.gpu‚77.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.91' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.97000000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç çQD÷3‚(:¹  ™¨‹Ó³ûÒ³ disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1510.in‚0* proc.memory.percent‚12.002197901407879 disk.drbd1356.in‚0 network.sent‚ 2.1500616e+07 network.recv‚ 1.601864e+06 disk.drbd2118.in‚0 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 2695.31640625 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 891.11328125 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.nvme0n1.in‚ 119.03125 disk.sda2.out‚ 891.11328125 disk.drbd1828.out‚0 disk.rbd1.out‚112.125 disk.sda1.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8674278259277 disk.drbd1550.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1965.out‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚55054.474609375 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1285.in‚0 cpu‚25.340645645043594 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.51430714388897 disk.drbd1584.in‚0 disk.drbd1510.out‚0# memory_percent‚89.0327294667562 disk.drbd2041.in‚0# proc.memory.rssMB‚ 2949.66015625 disk.drbd1817.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0! disk.dm-0.out‚55054.474609375 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0w®Žnø3‚(:’ ¨¨‹ÓåþÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚289.634' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.54466666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢d†×Kù3‚(:À  ¨¨‹Óà³á³ disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚55055.14111328125 disk.drbd1510.out‚0% disk./.usageGB‚1260.8674087524414 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2949.66015625 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd2118.out‚0* proc.memory.percent‚12.002197901407879 disk.drbd2078.in‚0 network.recv‚ 1.602821e+06 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.0390625 disk.dm-0.in‚ 119.03125 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0) disk./.usagePercent‚70.51430607719976 disk.drbd1552.out‚0 disk.nvme0n1.in‚ 119.03125 network.sent‚ 2.150368e+07 disk.drbd1510.in‚0 disk.sda2.out‚ 891.44140625 disk.drbd1931.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0( proc.memory.availableMB‚ 2414.4140625 disk.drbd1226.out‚0 disk.drbd2118.in‚0# disk.dm-0.out‚55055.14111328125 cpu‚25.340585041909044 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 891.44140625$ memory_percent‚90.17572402954102 disk.rbd1.in‚0 disk.rbd1.out‚ 112.3046875м¶Gú3‚(:’ ·¨‹Ó×ùÁ³ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.30733333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îºv"Mû3‚(:  ·¨‹ÓÆãá³ disk.drbd1356.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.out‚0 disk.sda2.out‚ 891.67578125 disk.drbd1285.out‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.sda2.in‚ 0.0390625 disk.drbd1965.out‚0# proc.memory.rssMB‚ 2949.66015625 disk.drbd1226.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚55055.60595703125 disk.rbd1.out‚ 112.4609375 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 2117.74609375 disk.drbd1289.out‚0& disk.nvme0n1.out‚55055.60595703125 disk.drbd1510.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2041.in‚0 cpu‚25.340569123455175 disk.drbd2118.in‚0$ memory_percent‚91.38286908467612 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0) disk./.usagePercent‚70.51433359778123 disk.sda.out‚ 891.67578125 disk.drbd1289.in‚0* proc.memory.percent‚12.002197901407879 network.sent‚ 2.1503746e+07 network.recv‚ 1.602887e+06% disk./.usageGB‚1260.8679008483887 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.0390625þÜåÞü3‚(:’ ƨ‹ÓØõ³³ gpu.0.gpu‚44.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.623' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.87433333333333 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÒ±½Cý3‚(:¸  ƨ‹ÓäûÔ³ disk.sda1.in‚0 disk.drbd1931.out‚0 disk.dm-0.out‚55056.10546875 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0* proc.memory.percent‚12.002197901407879 disk.drbd1552.in‚0 disk.sda2.in‚ 0.0390625$ memory_percent‚92.09252993265787# proc.memory.rssMB‚ 2949.66015625 disk.sda.in‚ 0.0390625 disk.sda.out‚ 892.0390625 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 1943.33984375 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.drbd2078.in‚0) disk./.usagePercent‚70.51432783765954 disk.rbd1.out‚ 112.640625# disk.nvme0n1.out‚55056.10546875 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 network.sent‚ 2.1574602e+07 disk.nvme0n1.in‚ 119.03125 disk.sda2.out‚ 892.0390625 disk.drbd1965.out‚0% disk./.usageGB‚1260.8677978515625 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 cpu‚25.34052023103824 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1356.out‚0 network.recv‚ 1.605959e+06 disk.drbd1289.out‚0 disk.drbd1552.out‚0 proc.cpu.threads‚174ܨ'4þ3/ _wandbruntime‚22535  _runtime‚22535ç'¯›Êÿ3Ä & lr-AdamW/pg1‚3.891431947088628e-06 & lr-AdamW/pg2‚3.891431947088628e-06 trainer/global_step‚12399 % _timestamp‚1.7848617717111502e+09 _runtime‚22535.068408945 _step‚1032$ílÏ€4É % _timestamp‚1.7848617717111502e+09 _runtime‚22535.068408945 _step‚1032 & lr-AdamW/pg1‚3.891431947088628e-06 & lr-AdamW/pg2‚3.891431947088628e-06 trainer/global_step‚12399ˆgËx˜44/ _wandbruntime‚22535  _runtime‚22535Qèzh$‚4”P ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3569824813195888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12941664973824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107690222 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3571233062176768 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3584174700977152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222854663 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222844593 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115164441 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12941664973824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚976818111507456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3584176072586752 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3584176072586752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12921340025490 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3584174700977152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚976818479598592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G?DeviceStatsMonitor.on_train_batch_start/inJõºè²active_split.all.peak‚660 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115154377 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70854964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33043956 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 _runtime‚22535.069868076 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3582744808610092Ðm.L(ƒ4™P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12943010409984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70855321 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 _step‚1033 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 trainer/global_step‚12399 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12943010409984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3571233036003328 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15824973734400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3571233062176768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3582746178649346 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3569824838623856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115154377 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚992643453332992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15824920370176 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107690216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115164441 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107690216 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222844593 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115164441 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12941664973824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚976818111507456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3584176072586752 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3584176072586752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12921340025490 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3584174700977152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚976818479598592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115154377 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70854964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33043956 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 _runtime‚22535.069868076 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3582744808610092 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3569824813195888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12941664973824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107690222 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3571233062176768 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3584174700977152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222854663 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70860530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37813939 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115173537 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12942687481856 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 trainer/global_step‚12399 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12922360827242 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222872126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3584456004392448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15826226795520 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1034 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107698589 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3583024631185508 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3571511971467264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107698589 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107698583 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚976897065964544 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15826171341312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12944032925184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3570103614969664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3584454632775680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37814287 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12921016215844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115163459 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222872126 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚976896697873408 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33046591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3570103640397632 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚22536.545889742 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107698583 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115163459 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3571511945293824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚992723292760064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3571511971467264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115173537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672Ú—Ä<''†4¡N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3584454632775680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37814287 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12921016215844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115163459 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚353 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222872126 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚976896697873408 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33046591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3570103640397632 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚22536.545889742 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107698583 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115163459 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3571511945293824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚992723292760064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3571511971467264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115173537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3571511945293824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222862042 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222862042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70860883 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12942687481856 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3584456004392448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12944032925184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70860530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37813939 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115173537 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12942687481856 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 trainer/global_step‚12399 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12922360827242 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222872126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3584456004392448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15826226795520 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1034 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107698589 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3583024631185508 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3571511971467264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107698589 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107698583 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚976897065964544 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15826171341312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12944032925184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3570103614969664Š—;z«4‡4/ _wandbruntime‚22536  _runtime‚225365± q ˆ4‡ 2loss/SRqqt1/assignment_loss‚0.4587004732869258 2loss/SRqqt2/assignment_loss‚0.4777225265483754 /loss/FBt2/detection_loss‚0.2715476500928998 &loss/total_loss‚7.3635121517102915 /loss/FRt1/detection_loss‚0.4953263423391043 0loss/FBt1/detection_loss‚0.49075798370034956 trainer/global_step‚12399 % _timestamp‚1.7848617732252316e+09 0loss/FRt2/assignment_loss‚0.6718819586035861 .loss/FRt2/detection_loss‚1.377848902925022 1loss/SRqqt2/detection_loss‚0.6032824049031551 _step‚1035 /loss/FRt1/assignment_loss‚1.607334051227843 2loss/FBt1/assignment_loss‚0.013151350118656554 2loss/FBt2/assignment_loss‚0.017839952849900673 1loss/SRqqt1/detection_loss‚0.8781185551144735 epoch‚44 _runtime‚22536.582426223ŠÐÇQ9‰4Œ 2loss/SRqqt1/assignment_loss‚0.4587004732869258 2loss/SRqqt2/assignment_loss‚0.4777225265483754 /loss/FBt2/detection_loss‚0.2715476500928998 &loss/total_loss‚7.3635121517102915 /loss/FRt1/detection_loss‚0.4953263423391043 0loss/FBt1/detection_loss‚0.49075798370034956 trainer/global_step‚12399 % _timestamp‚1.7848617732252316e+09 0loss/FRt2/assignment_loss‚0.6718819586035861 .loss/FRt2/detection_loss‚1.377848902925022 1loss/SRqqt2/detection_loss‚0.6032824049031551 _step‚1035 /loss/FRt1/assignment_loss‚1.60_›âÙ7334051227843 2loss/FBt1/assignment_loss‚0.013151350118656554 2loss/FBt2/assignment_loss‚0.017839952849900673 1loss/SRqqt1/detection_loss‚0.8781185551144735 epoch‚44 _runtime‚22536.582426223‹¬-ÚžŠ4‚(:“ Õ¨‹Ó±¢Þ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚308.804' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.93466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰QËB‹4‚(:·  Õ¨‹ÓÃÈ‚´ disk.rbd1.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 cpu‚25.34049248025932 disk.nvme0n1.in‚ 119.03125 disk.sda2.in‚ 0.0390625 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚12.002197901407879 network.sent‚ 2.1574668e+07 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.dm-0.out‚ 55056.3984375 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51432677097033 disk.rbd1.out‚ 112.703125 disk.drbd1285.in‚0" disk.nvme0n1.out‚ 55056.3984375 disk.drbd1226.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2041.out‚0% disk./.usageGB‚1260.8677787780762 disk.sda1.out‚0 disk.drbd1938.in‚0 network.recv‚ 1.606025e+06 disk.drbd1584.out‚0 disk.drbd2078.out‚0$ memory_percent‚92.09184646606445# proc.memory.rssMB‚ 2949.66015625 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 892.36328125 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.sda.in‚ 0.0390625 disk.sda2.out‚ 892.36328125 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0( proc.memory.availableMB‚ 1943.50781253X'ä“Œ4‚(:ˆ 䨋ÓÖþdz gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚302.256' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.752 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ‹ËJ4‚(:¿  䨋ÓÜÀç³ disk.dm-0.in‚ 119.03125" disk.dm-0.out‚55056.8486328125 disk.drbd2078.in‚0 disk.sda.in‚ 0.0390625 disk.sda.out‚ 892.70703125% disk.nvme0n1.out‚55056.8486328125 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 892.70703125$ memory_percent‚92.09116299947104% disk./.usageGB‚1260.8677787780762 disk.drbd1931.out‚0 disk.drbd2041.out‚0 network.sent‚ 2.1575744e+07 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 cpu‚25.340443957045963 disk.drbd2078.out‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚ 112.703125 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚12.002197901407879 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1550.out‚0) disk./.usagePercent‚70.51432677097033 disk.drbd2111.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2949.66015625 disk.drbd1965.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1943.67578125 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1828.in‚0 network.recv‚ 1.606942e+06 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0­;THŽ4‚(:½  ó¨‹ÓºÿÔ³ proc.cpu.threads‚17 network.sent‚ 2.157581e+07 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 893.06640625% disk.nvme0n1.out‚55057.2392578125 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0% proc.memory.availableMB‚ 1944.0625 cpu‚25.34041746520946 disk.drbd1226.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0% disk./.usageGB‚1260.8677787780762 disk.rbd1.out‚112.8369140625 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.sda.in‚ 0.0390625" disk.dm-0.out‚55057.2392578125 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0) disk./.usagePercent‚70.51432677097033 disk.drbd1289.in‚0 network.recv‚ 1.607008e+06 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0$ memory_percent‚92.08958943684897 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.drbd2041.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.sda.out‚ 893.06640625 disk.dm-0.in‚ 119.03125# proc.memory.rssMB‚ 2949.66015625* proc.memory.percent‚12.002197901407879 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.sda2.in‚ 0.0390625 disk.drbd1938.in‚0 disk.drbd1965.out‚0§3rž4‚(:“ ó¨‹ÓÄŠ·³ gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚310.804' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚103.60133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·’.ú4‚(:’ ‚©‹Ó°ÛÕ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.463' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.15433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_] ¼G‘4‚(:¼  ‚©‹Ó¶¸ô³ disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1552.in‚0 disk.sda2.in‚ 0.0390625 disk.drbd1931.in‚0 disk.drbd1356.in‚0$ memory_percent‚92.08860397338867* proc.memory.percent‚12.002197901407879 disk.drbd1828.in‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 1944.3046875 disk.rbd1.in‚0 disk.drbd1285.in‚0 network.recv‚ 1.608971e+06 cpu‚25.340356935927787 disk.rbd1.out‚112.8369140625 disk.drbd1584.out‚0 disk.drbd1965.in‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚55057.537109375 disk.drbd1584.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0# proc.memory.rssMB‚ 2949.66015625) disk./.usagePercent‚70.51432805099736 disk.drbd1550.in‚0 disk.drbd1226.out‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.sda.out‚ 893.421875 disk.drbd1938.in‚0! disk.dm-0.out‚55057.537109375 disk.drbd2041.out‚0 network.sent‚ 2.1637937e+07% disk./.usageGB‚1260.8678016662598 disk.sda.in‚ 0.0390625 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 893.421875 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd2041.in‚0K©A–ž’4‚(:“ ‘©‹Óľӳ gpu.0.gpu‚68.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚304.913' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.63766666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 IJ“4‚(:¿  ‘©‹ÓËÉí³$ disk.nvme0n1.out‚55057.712890625 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0! disk.dm-0.out‚55057.712890625 disk.drbd2078.out‚0 disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1289.out‚0# proc.memory.rssMB‚ 2949.66015625* proc.memory.percent‚12.002197901407879 network.sent‚ 2.1638003e+07 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda.in‚ 0.0390625 disk.sda.out‚ 893.75390625 disk.drbd1673.in‚0 network.recv‚ 1.609037e+06 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 cpu‚25.340319283152464 disk.drbd1584.in‚0 disk.drbd1938.in‚0( proc.memory.availableMB‚ 1943.8046875 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 893.75390625 disk.dm-0.in‚ 119.03125) disk./.usagePercent‚70.51430181044294 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚112.8994140625$ memory_percent‚92.09063847859701$ disk./.usageGB‚1260.867332458496 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1965.out‚0lc¶žJ”4‚(:¿   ©‹Ó†‰´ disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 network.recv‚ 1.611807e+06 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚55058.08740234375$ disk./.usageGB‚1260.867416381836 disk.drbd2078.in‚0 cpu‚25.3402589742611 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0$ memory_percent‚92.09067026774088 disk.drbd1552.out‚0) disk./.usagePercent‚70.51430650387545 disk.drbd1550.in‚0* proc.memory.percent‚12.002197901407879 proc.cpu.threads‚17 network.sent‚ 2.164947e+07 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 893.99609375 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1931.out‚0 disk.drbd1965.in‚0' proc.memory.availableMB‚ 1943.796875 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚112.8994140625 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2949.66015625 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2111.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚55058.08740234375 disk.sda.out‚ 893.99609375Õȱû•4‚(:’  ©‹Óùäè³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.673' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.55766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á£ù4–4/ _wandbruntime‚22621  _runtime‚22621±74 Ì—4Æ ' lr-AdamW/pg1‚3.6546860695059918e-06 ' lr-AdamW/pg2‚3.6546860695059918e-06 trainer/global_step‚12449 % _timestamp‚1.7848618586019967e+09 _runtime‚22621.959249748 _step‚1036ÿ@‹Ñ˜4Ë trainer/global_step‚12449 % _timestamp‚1.7848618586019967e+09 _runtime‚22621.959249748 _step‚1036 ' lr-AdamW/pg1‚3.6546860695059918e-06 ' lr-AdamW/pg2‚3.6546860695059918e-06Œ[Ê4™4/ _wandbruntime‚22621  _runtime‚22621˜~xM(š4”P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12994199966208 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115620393 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚996635025680896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3598172726667776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12992854530048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71134116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33175956 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223718895 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3598171355058176 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12994199966208 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115620393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3585178526701568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15887227784704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3583764927312656 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3598172726667776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37958165 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚996634604225536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12971099297012 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚12449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33175951 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚363 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115610329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3598171355058176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3585178526701568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37958528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 _step‚1037 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108108566 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115610329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚980747797896192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15887174420480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3583764901884688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚22621.963944422 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3585178500528128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108108572 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71134484 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108108566 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3596736001181700 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15887174420480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3583764901884688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚22621.963944422 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3585178500528128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108108572 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71134484 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108108566 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3596736001181700 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3598171355058176 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12994199966208 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115620393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3585178526701568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15887227784704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3583764927312656 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3598172726667776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37958165 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚996634604225536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12971099297012 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚12449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33175951 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 VGDeviceStatsMonitor.on_train_batch_start/active_byÏ ‰Pòtes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚363 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115610329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 '[4œ4/ _wandbruntime‚22623  _runtime‚22623Ý s#'4N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3584043729086432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12973464710050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108116933 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37961453 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37961096 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3584043703658464 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 _runtime‚22623.439892811 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚996714864044544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115629489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3585457409818624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚980826016171008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33178591 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12995222481408 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108116933 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3585457409818624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3597017193796482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12972120098652 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3598451286856704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15888424328192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3597015823757116 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784861860082241e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3598452658473472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3585457435992064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12993877038080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3585457435992064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15888479782400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115619411 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223746428 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3598452658473472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 _step‚1038 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223736344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71139687 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223746428 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3585457435992064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15888479782400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115619411 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223746428 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3598452658473472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 _step‚1038 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223736344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71139687 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223746428 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3584043729086432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12973464710050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108116933 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37961453 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37961096 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3584043703658464 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 _runtime‚22623.439892811 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚996714864044544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115629489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚357 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3585457409818624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚980826016171008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33178591 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12995222481408 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108116933 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3585457409818624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3597017193796482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12972120098652 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3598451286856704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15888424328192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3597015823757116 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784861860082241e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3598452658473472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3585457435992064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ŽþÞ4Ÿ4/ _wandbruntime‚22623  _runtime‚22623–Ù¿Œ 4‰ 2loss/SRqqt2/assignment_loss‚0.4970907263356644 1loss/SRqqt1/detection_loss‚0.9101776179857552 1loss/SRqqt2/detection_loss‚0.5324348299862431 $ _timestamp‚1.784861860115806e+09 _runtime‚22623.473052019 /loss/FRt1/assignment_loss‚1.683332768079665 0loss/FRt2/assignment_loss‚0.6404798341893602 2loss/SRqqt1/assignment_loss‚0.5620095621427446 4loss/FBt1/assignment_loss‚0.00025016495212315526 /loss/FRt2/detection_loss‚1.3838228864573885 /loss/FBt1/detection_loss‚0.6484913339518243 %loss/total_loss‚7.784399417759939 trainer/global_step‚12449 _step‚1039 0loss/FBt2/detection_loss‚0.39982213660455196 epoch‚44 3loss/FBt2/assignment_loss‚0.0032389859760526187 /loss/FRt1/detection_loss‚0.5232485710985662Øw¯¡4Ž 3loss/FBt2/assignment_loss‚0.0032389859760526187 /loss/FRt1/detection_loss‚0.5232485710985662 2loss/SRqqt2/assignment_loss‚0.4970907263356644 1loss/SRqqt1/detection_loss‚0.9101776179857552 1loss/SRqqt2/detection_loss‚0.5324348299862431 $ _timestamp‚1.784861860115806e+09 _runtime‚22623.473052019 /loss/FRt1/assignment_loss‚1.683332768079665 0loss/FRt2/assignment_loss‚0.6404798341893602 2loss/SRqqt1/assignment_loss‚0.5620095621427446 4loss/FBt1/assignment_loss‚0.00025016495212315526 /loss/FRt2/detection_loss‚1.3838228864573885 /loss/FBt1/detection_loss‚0.6484913339518243 %loss/total_loss‚7.784399417759939 trainer/global_step‚12449 _step‚1039 0loss/FBt2/detection_loss‚0.39982213660455196 epoch‚44å‘6ÍM¢4‚(:  ¯©‹ÓúÙû³ disk.nvme0n1.in‚ 119.03125 proc.cpu.threads‚17$ disk./.usageGB‚1260.867416381836$ memory_percent‚91.45943323771158 disk.sda.out‚ 894.49609375 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0( proc.memory.availableMB‚ 2098.9296875 network.recv‚ 1.612626e+06 disk.drbd1289.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2118.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 disk.drbd1550.out‚0 cpu‚25.340210521505014) disk./.usagePercent‚70.51430650387545 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd2111.in‚0 network.sent‚ 2.1651927e+07 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.388603846232098 disk.drbd1938.in‚0 disk.drbd2041.out‚0& disk.nvme0n1.out‚55058.32568359375 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1356.out‚0# proc.memory.rssMB‚ 2798.86328125 disk.sda2.out‚ 894.49609375 disk.dm-0.in‚ 119.03125 disk.drbd1828.in‚0 disk.drbd1510.out‚0# disk.dm-0.out‚55058.32568359375 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 113.033203125 disk.drbd1285.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚ 0.0390625eî02’£4‚(:‡ ¯©‹Óž§Þ³ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚297.432' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Zm›¤4‚(: ¾©‹Óڊг gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.651' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.88366666666667 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÅcJ¥4‚(:¿  ¾©‹ÓÅ‚ð³ disk.drbd1510.out‚0 disk.sda2.in‚ 0.0390625 disk.dm-0.in‚ 119.03125 disk.drbd1356.in‚0$ disk./.usageGB‚1260.867416381836 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0$ memory_percent‚91.45944913228354 disk.drbd1817.out‚0 disk.sda.out‚ 894.77734375) proc.memory.availableMB‚ 2098.92578125 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.drbd1552.in‚0$ disk.nvme0n1.out‚55058.642578125 disk.sda.in‚ 0.0390625 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd2111.out‚0 disk.drbd2118.in‚0! disk.dm-0.out‚55058.642578125 disk.drbd2078.out‚0 disk.rbd1.out‚ 113.033203125 disk.sda2.out‚ 894.77734375 network.sent‚ 2.1712589e+07 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0# proc.memory.rssMB‚ 2798.86328125 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1673.out‚0 disk.drbd1550.in‚0 disk.drbd1285.in‚0 cpu‚25.340173124159836) disk./.usagePercent‚70.51430650387545 disk.drbd1356.out‚0* proc.memory.percent‚11.388603846232098 network.recv‚ 1.614552e+06 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1828.in‚0Wž¦4‚(:“ Í©‹Óô›è³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚304.462' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚101.48733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²²EH§4‚(:½  Í©‹ÓÊñ‡´ disk.drbd2111.out‚0 disk.drbd1584.in‚0) disk./.usagePercent‚70.51430650387545 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd2118.in‚0 network.sent‚ 2.1715585e+07 disk.drbd1938.in‚0! disk.dm-0.out‚55058.802734375 disk.drbd1356.out‚0 disk.drbd2041.out‚0$ disk.nvme0n1.out‚55058.802734375 disk.drbd1226.out‚0 disk.drbd1673.out‚0$ disk./.usageGB‚1260.867416381836) proc.memory.availableMB‚ 2099.16796875* proc.memory.percent‚11.388603846232098 disk.sda2.in‚ 0.0390625 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.sda2.out‚ 895.0234375 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0390625 network.recv‚ 1.615642e+06 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda.out‚ 895.0234375 disk.drbd2078.in‚0 cpu‚25.340113097116735 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 113.095703125 disk.drbd1510.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2798.86328125 disk.drbd1289.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0$ memory_percent‚91.45846366882324 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1817.out‚0qùkýG¨4‚(:¼  Ü©‹ÓŠÜã³ disk.drbd1673.out‚0 disk.sda2.in‚ 0.0390625# proc.memory.rssMB‚ 2798.86328125 disk.drbd1289.in‚0 disk.drbd1289.out‚0$ disk./.usageGB‚1260.867416381836 disk.sda.in‚ 0.0390625 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1510.in‚0 disk.dm-0.out‚55059.21484375 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 2098.91015625 proc.cpu.threads‚17 network.sent‚ 2.1718453e+07 network.recv‚ 1.615882e+06 disk.drbd1584.out‚0# disk.nvme0n1.out‚55059.21484375 disk.drbd1828.in‚0$ memory_percent‚91.45951271057129 disk.sda1.out‚0 disk.drbd2118.in‚0 cpu‚25.34006525174451 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.388603846232098 disk.drbd1552.in‚0 disk.sda.out‚ 895.24609375 disk.drbd1938.out‚0 disk.rbd1.out‚ 113.095703125) disk./.usagePercent‚70.51430650387545 disk.sda2.out‚ 895.24609375 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0fKâT©4‚(:’ Ü©‹Óæ࿳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.785' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.26166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÊÓòª4‚(:’ ë©‹ÓÏã³ gpu.0.gpu‚71.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.531' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.84366666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šH–ÚG«4‚(:¼  ë©‹ÓýÃþ³# disk.nvme0n1.out‚55059.65234375 disk.drbd1289.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2798.86328125 disk.drbd2111.in‚0$ memory_percent‚91.45706494649252 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.sda.in‚ 0.0390625* proc.memory.percent‚11.388603846232098 disk.drbd1285.out‚0 disk.drbd1550.in‚0 disk.dm-0.out‚55059.65234375 disk.drbd1828.in‚0 disk.rbd1.out‚113.2294921875) proc.memory.availableMB‚ 2099.51171875 disk.sda.out‚ 895.7421875 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51430650387545 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 cpu‚25.340060325241677 network.recv‚ 1.615948e+06$ disk./.usageGB‚1260.867416381836 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1226.out‚0 disk.sda2.in‚ 0.0390625 network.sent‚ 2.1718519e+07 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 895.7421875 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0ûqÒ*4¬4/ _wandbruntime‚22708  _runtime‚22708:XÁ: ­4Ä tra >ý½iner/global_step‚12499 % _timestamp‚1.7848619452869492e+09 _runtime‚22708.644181132 _step‚1040 & lr-AdamW/pg1‚3.425137611993407e-06 & lr-AdamW/pg2‚3.425137611993407e-06ÿ§.£Ï®4É & lr-AdamW/pg1‚3.425137611993407e-06 & lr-AdamW/pg2‚3.425137611993407e-06 trainer/global_step‚12499 % _timestamp‚1.7848619452869492e+09 _runtime‚22708.644181132 _step‚1040cà¯4¯4/ _wandbruntime‚22708  _runtime‚22708ñdÊõ(°4–P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224603267 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚372 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13045389522432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3599123991226368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224593197 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71413447 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3610728563792562 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116066281 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚984677116193792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3612168009139200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15949518012928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108526922 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3597705016001456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3612168009139200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13044044086272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13022203179820öW¹’!(±4›P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224593197 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71413447 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3610728563792562 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116066281 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚984677116193792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3612168009139200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15949518012928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108526922 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3597705016001456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3612168009139200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13044044086272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13022203179820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224603267 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚372 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13045389522432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3599123991226368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006‘²0Ö²4‚(:’ ú©‹Ó ëí³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚292.463' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.48766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à„I]I³4‚(:¾  ú©‹ÓÛ—‹´ disk.drbd1550.out‚0% proc.memory.availableMB‚ 2099.6875 disk.rbd1.out‚113.2294921875 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 network.recv‚ 1.619564e+06& disk.nvme0n1.out‚55060.11962890625 disk.drbd1552.out‚0# disk.dm-0.out‚55060.11962890625 disk.drbd1965.in‚0 disk.drbd1828.out‚0 cpu‚25.34001168910056 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.sda.out‚ 895.98046875 disk.drbd1817.in‚0* proc.memory.percent‚11.388603846232098 disk.sda.in‚ 0.0390625 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 895.98046875 disk.drbd1356.out‚0) disk./.usagePercent‚70.51431866413238 disk.drbd1673.in‚0 disk.sda2.in‚ 0.0390625 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd2041.in‚0 network.sent‚ 2.1760747e+07 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1289.in‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.dm-0.in‚ 119.03125# proc.memory.rssMB‚ 2798.86328125 disk.drbd1817.out‚0# disk./.usageGB‚1260.86763381958 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0$ memory_percent‚91.45634969075522'~ý4´4/ _wandbruntime‚22710  _runtime‚22710hߪ%µ4 N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15950808104960 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3611008386368090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71418595 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71418978 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3597983792347264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚12499 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116075363 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚378 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚383 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3612447940937728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116075363 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13023223981460 _runtime‚22710.123941558 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚984755702559744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224610646 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3612449312554496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 _step‚1042 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13046412037632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108535283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚984755334468608 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3599402874343424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3599402900516864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116085441 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3599402874343424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13045066594304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3597983817775232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224610646 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13046412037632 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3612449312554496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38108004 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7848619467663014e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33310591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224620730 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3599402900516864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224620730 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108535283 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33310596 TFDeviceStat7gz|sMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084rŒðŽ+'¶4¥N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13045066594304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108535289 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1000706085022208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33310596 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15950808104960 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3611008386368090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71418595 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71418978 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3597983792347264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚12499 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116075363 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚378 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚383 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3612447940937728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116075363 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13023223981460 _runtime‚22710.123941558 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚984755702559744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224610646 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3612449312554496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 _step‚1042 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13046412037632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108535283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚984755334468608 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3599402874343424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3599402900516864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116085441 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3599402874343424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13045066594304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3597983817775232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224610646 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13046412037632 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3612449312554496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38108004 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7848619467663014e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33310591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224620730 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3599402900516864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224620730 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108535283 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3626164663220224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3613069429577728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13096579078656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38253263 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108945266 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3626166034829824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13074651673914 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 % _timestamp‚1.7848620320374715e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3624718386324916 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3611645104690256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108945266 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚988606434491392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108945272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71693219 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33439956 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 _step‚1045 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108945272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225477569 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13096579078656 KEDeviceSta}†a8tsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664e©/!(É4›P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116522233 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38252886 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33439951 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3626164663220224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3613069429577728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13096579078656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38253263 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108945266 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3626166034829824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13074651673914 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 % _timestamp‚1.7848620320374715e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3624718386324916 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3611645104690256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108945266 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚988606434491392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108945272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71693219 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33439956 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 _step‚1045 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108945272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225477569 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13096579078656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825•vŽ!Ì4Ê4/ _wandbruntime‚22796  _runtime‚22796sxð$'Ë4žN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3626444595018752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚379 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1004698376461312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 $ _timestamp‚1.784862033514986e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13097601593856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71698772 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3613348365041664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225484948 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16013355603968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 trainer/global_step‚12549 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33442591 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13074327864268 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16013295955456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3613348338868224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13096256150528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33442596 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1004697948721664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3626445966635520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71698393 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38256176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3624998208900332 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116531315 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38255802 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3626445966635520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3613348365041664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225495032 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3611923906464032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 _step‚1046 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225495032 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3611923881036064 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116541393 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3611923906464032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 _step‚1046 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225495032 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3611923881036064 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116541393 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13096256150528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33442596 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1004697948721664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3626445966635520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71698393 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38256176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3624998208900332 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116531315 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38255802 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3626445966635520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3613348365041664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225495032 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16074549910528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16074498643456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13124410945436 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3640161317301248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13146423198720 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13146423198720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚377 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚992535752788992 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71973407 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3638709578896524 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3640161317301248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33571951 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3638710948935778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226351871 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1008609883341312 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116988249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33571956 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3625585193379056 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3625585167951088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13125755556722 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109363616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚992535384697856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7848621190742831e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109363616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚372 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3627014920275968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3627014894102528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38401079 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116988249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38401451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3627014920275968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116978185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71973030 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13147768634880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 _runtime‚22882.434921907 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3640162688910848 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109363622 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109363622 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚12599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226341801 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116978185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1008610302699520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 17301504000ü]‰'á4›P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116988249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33571956 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3625585193379056 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3625585167951088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13125755556722 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109363616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚992535384697856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7848621190742831e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109363616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚372 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3627014920275968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3627014894102528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38401079 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116988249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38401451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3627014920275968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116978185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71973030 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13147768634880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 _runtime‚22882.434921907 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3640162688910848 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109363622 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109363622 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚12599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226341801 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116978185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1008610302699520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13147768634880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16074549910528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16074498643456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13124410945436 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3640161317301248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13146423198720 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13146423198720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚377 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚992535752788992 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71973407 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3638709578896524 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3640161317301248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33571951 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3638710948935778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_—Uø˜train_batch_start/allocation.all.allocated‚ 226351871 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1008609883341312™ô­Ž4â4/ _wandbruntime‚22883  _runtime‚22883›Í¾Ã%'ã4ŸN YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13147445706752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226359250 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13147445706752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109371983 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3638989401471940 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3625863995152832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226369334 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚364 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3638990771511306 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226359250 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3640441249099776 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3625863969724864 _step‚1050 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116997345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3627293803393024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16075806794752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3640442620716544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13125431747076 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116987267 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71978640Į́Œ*'ä4¤N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109371983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71979009 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33574591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116987267 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3640442620716544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚992614339154944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚22883.90688762 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3627293803393024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1008689730792960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13148791150080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13148791150080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3625863969724864 _step‚1050 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116997345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3627293803393024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16075806794752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3640442620716544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13125431747076 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116987267 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71978640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13147445706752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226359250 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13147445706752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109371983 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3638989401471940 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3625863995152832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226369334 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚364 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3638990771511306 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226359250 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3640441249099776 F&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6392722129821777 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027401240542531013 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8555733561515808 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2995818555355072 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17940953373908997 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7302624583244324 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28861004114151 epoch‚44 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6643799543380737 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08803720772266388 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6388697028160095 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6510542631149292 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6932092308998108 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1441502869129181 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6826454401016235 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6596602201461792 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 _step‚1052 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6510542631149292 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7107214331626892 *jet/accuracy_2_of_2‚0.1739431917667389 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3918841779232025 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 %particle/f_score‚0.65617835521698 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3750848174095154 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5195205211639404 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 6validation_average_jet_accuracy‚0.4098312556743622 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6451652646064758 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7234966158866882 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03240741416811943 ,jet/accuracy_4_of_4‚0.018756305798888206 % _timestamp‚1.7848621401295366e+09 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6413387656211853 *jet/accuracy_1_of_1‚0.3763340413570404 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.36153197288513184 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1750321388244629 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7165890336036682 *jet/accuracy_2_of_3‚0.6110299229621887 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6469070315361023 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.28424498438835144 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 *jet/accuracy_2_of_4‚0.8373957276344299 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.808952808380127 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6714876890182495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3736865520477295 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6424232721328735 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034577082842588425 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.361570805311203 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38739830255508423 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.554084300994873 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4568575918674469 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 +jet/accuracy_3_of_4‚0.10971427708864212 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9241049885749817 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.803146243095398 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3525543510913849 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purõ@Œàity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3120475709438324 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19146953523159027 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.36153197288513184 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2800453305244446 *jet/accuracy_1_of_2‚0.7939143180847168 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35950708389282227 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028983155265450478 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 *jet/accuracy_1_of_3‚0.9394849538803101 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.705841891046064 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38915786147117615 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6965377926826477 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7170947194099426 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27295929193496704 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6695687174797058 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.881477952003479 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7678483724594116 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7766839861869812 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5297715067863464 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 _runtime‚22903.487631163 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16160444915294647 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.41034644842147827 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22599291801452637 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38985633850097656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.314432829618454 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9832039475440979 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7216353416442871 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5831097960472107 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.761881411075592 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05475744605064392 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30285149812698364 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746458292007446 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2743197977542877 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3728514015674591 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04553152993321419 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07176747173070908 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.781926691532135 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 +particle/specificity‚0.9137938618659973 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17399659752845764 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22952386736869812 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6986857056617737 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 trainer/global_step‚12599 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7249317169189453 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6979913711547852 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8007562756538391 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699338793754578 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2930779755115509 /particle/accuracy_4_of_4‚0.9640940427780151 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8013328313827515 +jet/accuracy_3_of_3‚0.07885482162237167 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509YXd¡;ì4Žv =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22952386736869812 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6986857056617737 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 trainer/global_step‚12599 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7249317169189453 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6979913711547852 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8007562756538391 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699338793754578 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2930779755115509 /particle/accuracy_4_of_4‚0.9640940427780151 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8013328313827515 +jet/accuracy_3_of_3‚0.07885482162237167 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3602948784828186 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6392722129821777 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027401240542531013 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8555733561515808 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2995818555355072 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17940953373908997 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7302624583244324 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28861004114151 epoch‚44 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6643799543380737 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08803720772266388 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6388697028160095 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6510542631149292 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6932092308998108 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1441502869129181 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6826454401016235 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6596602201461792 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 _step‚1052 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6510542631149292 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7107214331626892 *jet/accuracy_2_of_2‚0.1739431917667389 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3918841779232025 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7846153974533081 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 %particle/f_score‚0.65617835521698 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3750848174095154 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5195205211639404 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 6validation_average_jet_accuracy‚0.4098312556743622 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6451652646064758 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7234966158866882 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03240741416811943 ,jet/accuracy_4_of_4‚0.018756305798888206 % _timestamp‚1.7848621401295366e+09 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6413387656211853 *jet/accuracy_1_of_1‚0.3763340413570404 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.36153197288513184 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1750321388244629 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7165890336036682 *jet/accuracy_2_of_3‚0.6110299229621887 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6469070315361023 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.28424498438835144 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 *jet/accuracy_2_of_4‚0.8373957276344299 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.808952808380127 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6714876890182495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3736865520477295 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6424232721328735 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034577082842588425 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.361570805311203 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38739830255508423 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.554084300994873 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4568575918674469 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6285714507102966 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 +jet/accuracy_3_of_4‚0.10971427708864212 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9241049885749817 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.803146243095398 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3525543510913849 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3120475709438324 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19146953523159027 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.36153197288513184 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2800453305244446 *jet/accuracy_1_of_2‚0.7939143180847168 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35950708389282227 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028983155265450478 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 *jet/accuracy_1_of_3‚0.9394849538803101 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.705841891046064 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38915786147117615 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6965377926826477 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7170947194099426 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27295929193496704 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6695687174797058 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.881477952003479 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7678483724594116 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7766839861869812 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5297715067863464 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 _runtime‚22903.487631163 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16160444915294647 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.41034644842147827 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22599291801452637 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38985633850097656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.314432829618454 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9832039475440979 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7216353416442871 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5831097960472107 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.761881411075592 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05475744605064392 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30285149812698364 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8746458292007446 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2743197977542877 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3728514015674591 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04553152993321419 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07176747173070908 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.781926691532135 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 +particle/specificity‚0.9137938618659973 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17399659752845764 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552œùVÄg›í4‚(: ½«‹Óñ©Ô³ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚58.0 gpu.0.powerWatts‚106.061' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.35366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›[lóKî4‚(:À  ½«‹Óª—÷³ disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0# proc.memory.rssMB‚ 3837.73828125# disk.dm-0.out‚55066.55908203125 disk.drbd2111.out‚0) proc.memory.availableMB‚ 1056.83984375 network.sent‚ 2.1910766e+07$ disk./.usageGB‚1260.869441986084 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0 disk.drbd1931.out‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 network.recv‚ 1.632609e+06 disk.sda2.in‚ 0.0390625 disk.rbd1.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0 disk.sda.in‚ 0.0390625 disk.sda.out‚ 900.9921875 disk.drbd1673.out‚0 disk.rbd1.out‚ 114.318359375 disk.drbd1226.in‚0 disk.sda2.out‚ 900.9921875 disk.drbd1938.out‚0& disk.nvme0n1.out‚55066.55908203125$ memory_percent‚95.69970766703287) disk./.usagePercent‚70.51441978626893 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 119.03125 cpu‚25.341470189373627 disk.dm-0.in‚ 119.03125 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0) proc.memory.percent‚15.61579704284668ºÿ`Üœï4‚(:‘ Ì«‹Ó±µ³ gpu.0.gpu‚64.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚289.07' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.35666666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú”è Oð4‚(:Ä  Ì«‹ÓÐùÓ³# proc.memory.rssMB‚ 2889.19140625 disk.drbd1289.out‚0 network.sent‚ 2.1910874e+07 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 2914.62890625 disk.rbd1.out‚ 114.318359375 disk.drbd1226.out‚0 disk.drbd1552.in‚0# disk.dm-0.out‚56036.35205078125 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0% disk./.usageGB‚1260.8732223510742 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.sda2.out‚ 901.32421875 disk.sda2.in‚ 0.0390625* proc.memory.percent‚11.756149927775066 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.0390625 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 network.recv‚ 1.632717e+06 disk.drbd1356.in‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51463120406922 disk.drbd1965.in‚0& disk.nvme0n1.out‚56036.35205078125 disk.drbd1584.out‚0 disk.sda.out‚ 901.32421875$ memory_percent‚88.14034461975098 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 cpu‚25.340877359355556 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0õ€s’ñ4‚(:‡ Û«‹ÓìÄÚ³ gpu.0.gpu‚57.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚59.0 gpu.0.powerWatts‚107.166' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚35.722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É \_Nò4‚(:à  Û«‹Ó­Úü³ disk.drbd1289.out‚0 disk.rbd1.out‚114.4521484375 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.sda.in‚ 0.0390625 disk.drbd1931.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 cpu‚25.34082816150565% disk./.usageGB‚1260.8732223510742 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.sda.out‚ 901.68359375 disk.drbd1965.out‚0 disk.drbd1938.out‚0 disk.rbd1.in‚0 network.sent‚ 2.1911358e+07 disk.sda2.in‚ 0.0390625# disk.dm-0.out‚56036.76220703125# proc.memory.rssMB‚ 2889.19140625 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1289.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 2915.7421875) disk./.usagePercent‚70.51463120406922 disk.drbd2118.in‚0 disk.drbd1673.out‚0& disk.nvme0n1.out‚56036.76220703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 901.68359375 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1285.in‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 network.recv‚ 1.633702e+06 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0$ memory_percent‚88.13581466674805* proc.memory.percent‚11.756149927775066þ<^Ûó4‚(:’ ê«‹Ó ƒ»³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚264.766' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.25533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ6ýNHô4‚(:½  ê«‹ÓҲݳ disk.dm-0.in‚ 119.03125 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2889.19140625% disk./.usageGB‚1260.8694801330566 disk.drbd1510.out‚0 disk.drbd1356.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0$ memory_percent‚88.13578287760416 disk.drbd1226.in‚0# disk.dm-0.out‚57007.06103515625) disk./.usagePercent‚70.51442191964735 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 902.0546875 disk.drbd2078.out‚0 disk.drbd2111.in‚0* proc.memory.percent‚11.756149927775066 cpu‚25.340791206738896 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.rbd1.in‚0 network.sent‚ 2.1992282e+07 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚ 0.0390625 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125& disk.nvme0n1.out‚57007.06103515625 disk.sda1.out‚0 disk.drbd1673.out‚0 network.recv‚ 1.635902e+06 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚114.4521484375 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 902.0546875 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0# proc.memory.availableMB‚2915.75 disk.sda.in‚ 0.0390625 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0ç õ4‚(:’ ù«‹Óâû³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.238' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.07933333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y”Œ¢Eö4‚(:º  ù«‹Óïíæ³$ memory_percent‚88.13581466674805 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.756149927775066 disk.drbd1584.out‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.0390625 disk.sda2.out‚902.8125 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 network.sent‚ 2.199239e+07 disk.sda.in‚ 0.0390625 disk.drbd1289.in‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1931.out‚0 disk.drbd2041.in‚0) disk./.usagePercent‚70.51442191964735 disk.drbd1285.in‚0 disk.dm-0.in‚ 119.03125 cpu‚25.340742689659827 disk.drbd1584.in‚0# disk.dm-0.out‚57007.24072265625 network.recv‚ 1.63601e+06 disk.drbd2118.in‚0( proc.memory.availableMB‚ 2915.7421875 disk.drbd1356.out‚0 disk.drbd2111.in‚0# proc.memory.rssMB‚ 2889.19140625 disk.rbd1.out‚114.5146484375 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8694801330566 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚57007.24072265625 disk.sda.out‚902.8125 disk.drbd2041.out‚02ü÷4‚(:’ ˆ¬‹Ó“²»³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚295.604' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.53466666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q/†•ø4‚(:½  ˆ¬‹ÓíÅܳ disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚57008.10693359375 disk.drbd1550.out‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 119.03125 disk.drbd1552.in‚0 disk.sda2.out‚ 903.046875 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1584.out‚0$ memory_percent‚88.13581466674805 disk.drbd1226.out‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.756149927775066 disk.drbd1673.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0# disk.dm-0.out‚57008.10693359375 disknÎ|v³.drbd1828.in‚0 disk.rbd1.out‚114.5146484375 disk.nvme0n1.in‚ 119.03125 disk.drbd1510.in‚0 network.recv‚ 1.63601e+06 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1356.in‚0 disk.drbd1552.out‚0% disk./.usageGB‚1260.8694229125977 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.out‚ 903.046875 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 cpu‚25.34068372737143 disk.drbd1356.out‚0 disk.drbd1931.in‚0( proc.memory.availableMB‚ 2915.7421875 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda.in‚ 0.0390625 network.sent‚ 2.199239e+07 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0) disk./.usagePercent‚70.51441871957974# proc.memory.rssMB‚ 2889.19140625ˆâ(š4ù4/ _wandbruntime‚22994  _runtime‚22994Že—ÄÌú4Æ ' lr-AdamW/pg1‚2.7799729046878937e-06 ' lr-AdamW/pg2‚2.7799729046878937e-06 trainer/global_step‚12649 % _timestamp‚1.7848622306574135e+09 _runtime‚22994.014677355 _step‚1053JyŽäÑû4Ë _runtime‚22994.014677355 _step‚1053 ' lr-AdamW/pg1‚2.7799729046878937e-06 ' lr-AdamW/pg2‚2.7799729046878937e-06 trainer/global_step‚12649 % _timestamp‚1.7848622306574135e+09ÕÓÎ4ü4/ _wandbruntime‚22994  _runtime‚22994Wé±H(ý4–P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109871404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 trainer/global_step‚12649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3656724266546176 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117491513 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 _step‚1054 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚996514070599680 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3642084283219024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72288766 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13182468278844 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3643521035352064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117491513 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3656725638155776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117481449 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1012658841261056 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33710848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33710843 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72288414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109871398 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3656725638155776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚996513702508544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109871398 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13204576630272 % _timestamp‚1.7848622306585495e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3643521035352064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16144770661376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3655265381458614 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13203231194112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227362917 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚347 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13203231194112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227352847 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3643521061525504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16144721491456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3655266751497868 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109871404 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3656724266546176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3643521061525504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1012658424000000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 13716096006âÞz!(þ4›P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109871398 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3656725638155776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚996513702508544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109871398 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13204576630272 % _timestamp‚1.7848622306585495e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3643521035352064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16144770661376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3655265381458614 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13203231194112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227362917 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚347 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13203231194112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227352847 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3643521061525504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16144721491456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3655266751497868 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109871404 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3656724266546176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3643521061525504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1012658424000000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109871404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 trainer/global_step‚12649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3656724266546176 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117491513 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 _step‚1054 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚996514070599680 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3642084283219024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72288766 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13182468278844 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3643521035352064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117491513 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3656725638155776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117481449 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1012658841261056 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33710848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33710843 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72288414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254ž¡4Ì;ÿ4‚(:’ —¬‹ÓÎàƳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚269.174' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.72466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À\YŒC€5‚(:¸  —¬‹Ó½Žä³ disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2889.19140625 proc.cpu.threads‚17$ disk./.usageGB‚1260.869441986084 disk.drbd1584.out‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.756149927775066# disk.dm-0.out‚57008.32177734375 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 903.453125 disk.drbd1938.in‚0 disk.drbd2041.in‚0$ memory_percent‚88.13591003417969 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 119.03125 cpu‚25.3406571002831 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.rbd1.out‚ 114.6484375) disk./.usagePercent‚70.51441978626893 disk.drbd1817.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚57008.32177734375 disk.sda2.in‚ 0.0390625 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 119.03125 network.recv‚ 1.637102e+06 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 903.453125 disk.drbd1931.out‚0 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1510.out‚0 network.sent‚ 2.1995443e+07 disk.sda1.out‚0 disk.sda.in‚ 0.0390625 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0& proc.memory.availableMB‚ 2915.71875 disk.drbd2118.out‚0MÎJx45/ _wandbruntime‚22995  _runtime‚22995.f”Kª!‚5ŸN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227380380 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3657004198344704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3655545204034030 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38580453 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚996592288874496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72293936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚996592656965632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109879771 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117490531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33713483 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784862232135047e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109879771 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227370296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38580805 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3642363084992800 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3643799970816000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3643799944642560 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117500609 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13205599145472 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3642363059564832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227370296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227380380 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1012738257206784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3655546574073396 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3657005569961472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16145968332288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16146021689344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109879765 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚1320425370214ŽQ¥-{4 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3643799944642560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328“;ñ*'ƒ5¤N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109879771 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117490531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33713483 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784862232135047e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109879771 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227370296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38580805 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3642363084992800 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3643799970816000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3643799944642560 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117500609 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13205599145472 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3642363059564832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227370296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227380380 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1012738257206784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3655546574073396 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3657005569961472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16145968332288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16146021689344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109879765 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13204253702144 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3643799944642560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227380380 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3657004198344704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3655545204034030 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38580453 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚996592288874496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72293936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚996592656965632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Ÿÿm,4„5/ _wandbruntime‚22995  _runtime‚22995•j¶Ê …5‡ _runtime‚22995.52864869 $loss/total_loss‚7.29046615914407 % _timestamp‚1.7848622321714017e+09 0loss/FRt1/assignment_loss‚1.6141306646750309 0loss/FBt2/detection_loss‚0.42202413460312027 trainer/global_step‚12649 _step‚1056 0loss/FRt2/assignment_loss‚0.5879861855844282 2loss/SRqqt1/assignment_loss‚0.5109949669544688 2loss/SRqqt2/assignment_loss‚0.4465955532418775 3loss/FBt1/assignment_loss‚0.0011345214042961516 2loss/FBt2/assignment_loss‚0.007222259975918877 /loss/FRt2/detection_loss‚1.3973269854449306 0loss/SRqqt1/detection_loss‚0.772562893154402 1loss/SRqqt2/detection_loss‚0.5910661771236079 /loss/FRt1/detection_loss‚0.5253350911863475 0loss/FBt1/detection_loss‚0.41408672579564154 epoch‚45€6$G†5Œ 2loss/SRqqt1/assignment_loss‚0.5109949669544688 2loss/SRqqt2/assignment_loss‚0.4465955532418775 3loss/FBt1/assignment_loss‚0.0011345214042961516 2loss/FBt2/assignment_loss‚0.007222259975918877 /loss/FRt2/detection_loss‚1.3973269854449306 0loss/SRqqt1/detection_loss‚0.772562893154402 1loss/SRqqt2/detection_loss‚0.5910661771236079 /loss/FRt1/detection_loss‚0.5253350911863475 0loss/FBt1/detection_loss‚0.41408672579564154 epoch‚45 _runtime‚22995.52864869 $loss/total_loss‚7.29046615914407 % _timestamp‚1.7848622321714017e+09 0loss/FRt1/assignment_loss‚1.6141306646750309 0loss/FBt2/detection_loss‚0.42202413460312027 trainer/global_step‚12649 _step‚1056 0loss/FRt2/assignment_loss‚0.5879861855844282  P‡5‚(:’ ¦¬‹ÓÈËѳ gpu.0.gpu‚69.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.052' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.01733333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-¡†”@ˆ5‚(:µ  ¦¬‹ÓÞŽð³ disk.drbd1584.out‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.756149927775066 disk.drbd2118.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1938.out‚0 disk.sda.out‚ 903.828125 disk.dm-0.in‚ 119.03125 disk.drbd1550.in‚0 network.recv‚ 1.637252e+06 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 network.sent‚ 2.1995593e+07( proc.memory.availableMB‚ 2915.9296875 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0$ memory_percent‚88.13505172729492 disk.drbd1673.out‚0 disk.sda2.out‚ 903.828125 disk.drbd2111.in‚0 disk.drbd1510.out‚0) disk./.usagePercent‚70.51441871957974 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0# disk.nvme0n1.out‚57008.85546875 disk.drbd2111.out‚0 cpu‚25.3406087176083 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1552.in‚0 disk.drbd1673.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0% disk./.usageGB‚1260.8694229125977 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.dm-0.out‚57008.85546875 disk.drbd1965.in‚0 disk.rbd1.out‚ 114.6484375# proc.memory.rssMB‚ 2889.19140625 disk.drbd1552.out‚0x<ƒÖ‰5‚(:’ µ¬‹Ó½Ï׳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚268.986' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.66199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E †x>Š5‚(:³  µ¬‹ÓÍ‹÷³) disk./.usagePercent‚70.51441871957974 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0( proc.memory.availableMB‚ 2915.6796875 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.sda2.out‚904.3125 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 network.sent‚ 2.1995659e+07 disk.sda.in‚ 0.0390625 disk.drbd1552.out‚0 disk.sda.out‚904.3125 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 114.7109375 cpu‚25.340538803176663 disk.sda2.in‚ 0.0390625 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1828.in‚0% disk./.usageGB‚1260.8694229125977 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.dm-0.out‚57009.02734375 disk.drbd1356.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 119.03125 disk.nvme0n1.in‚ 119.03125# disk.nvme0n1.out‚57009.02734375* proc.memory.percent‚11.756149927775066 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 network.recv‚ 1.637318e+06 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1673.in‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2889.19140625 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd2041.out‚0 disk.drbd1938.in‚0$ memory_percent‚88.13606897989908´CH‹5‚(:½  Ĭ‹Ó¥±æ³# memory_percent‚88.1370703379313 disk.sda.out‚ 904.515625 disk.sda2.out‚ 904.515625 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0# proc.memory.rssMB‚ 2889.19140625 network.recv‚ 1.637384e+06% disk./.usageGB‚1260.8657608032227 disk.nvme0n1.in‚ 119.03125 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.0390625 disk.dm-0.in‚ 119.03125 disk.drbd2041.in‚0 network.sent‚ 2.1995725e+07 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0# disk.dm-0.out‚57009.37451171875& disk.nvme0n1.out‚57009.37451171875 proc.cpu.threads‚17 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 114.7109375 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1226.out‚0 cpu‚25.340501927393113 disk.drbd1510.out‚0 disk.sda.in‚ 0.0390625 disk.drbd2111.out‚0) proc.memory.availableMB‚ 2915.43359375 disk.drbd1828.out‚0 disk.drbd1226.in‚0) disk./.usagePercent‚70.51421391525251 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0* proc.memory.percent‚11.756149927775066 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0–ëï’Œ5‚(:‡ Ĭ‹Ó¢‘̳ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.169' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.723 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00†ÁBI5‚(:¾  Ó¬‹Óåšô³ disk.drbd1285.out‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 119.03125 disk.drbd1356.in‚0& disk.nvme0n1.out‚57010.60498046875 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1356.out‚0 cpu‚25.340464527242066 network.sent‚ 2.1998796e+07) disk./.usagePercent‚70.51421391525251 disk.drbd1673.out‚0 disk.sda2.out‚ 904.87109375 disk.drbd1965.in‚0 disk.rbd1.out‚114.875 disk.drbd1226.in‚0 disk.sda.out‚ 904.87109375 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.756149927775066$ memory_percent‚88.13557624816895) proc.memory.availableMB‚ 2915.80078125 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0% disk./.usageGB‚1260.8657608032227 disk.drbd1510.out‚0 network.recv‚ 1.638346e+06 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1289.in‚0# disk.dm-0.out‚57010.60498046875 disk.drbd2078.in‚0# proc.memory.rssMB‚ 2889.19140625 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd2078.out‚0™^릞Ž5‚(:“ Ó¬‹Óðâѳ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚330.003' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚110.00099999999999 gpu.0.smClock‚1365 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©"1 5‚(:’ ⬋ÓÉéä³ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚279.301' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.10033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ŠÕH5‚(:½  ⬋ӧ芴 disk.dm-0.in‚ 119.03125 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 119.03125 proc.cpu.threads‚17 network.sent‚ 2.2077375e+07 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0# proc.memory.rssMB‚ 2889.19140625 disk.drbd1584.in‚0( proc.memory.availableMB‚ 2915.8046875* proc.memory.percent‚11.756149927775066 disk.drbd2078.out‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.8657836914062 disk.drbd1552.out‚0 disk.sda.out‚ 905.3359375 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1510.in‚0 cpu‚25.340405387979242 disk.drbd1673.out‚0 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 network.recv‚ 1.641318e+06 disk.drbd1931.out‚0 disk.rbd1.out‚ 115.0546875 disk.drbd1510.out‚0 disk.sda.in‚ 0.0390625 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚57011.0068359375 disk.drbd1550.in‚0" disk.dm-0.out‚57011.0068359375 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.sda2.out‚ 905.3359375 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0$ memory_percent‚88.13556035359701) disk./.usagePercent‚70.51421519527956 disk.sda2.in‚ 0.0390625 disk.drbd1828.out‚0¢yF34‘5/ _wandbruntime‚23080  _runtime‚23080b¤ Ì’5Æ ' lr-AdamW/pg1‚2.5795055203092294e-06 ' lr-AdamW/pg2‚2.5795055203092294e-06 trainer/global_step‚12699 % _timestamp‚1.7848623175809138e+09 _runtime‚23080.938161327 _step‚1057Ó"â5Ñ“5Ë ' lr-AdamW/pg2‚2.5795055203092294e-06 trainer/global_step‚12699 % _timestamp‚1.7848623175809138e+09 _runtime‚23080.938161327 _step‚1057 ' lr-AdamW/pg1‚2.5795055203092294e-06¡ÿHag4”5/ _wandbruntime‚23080  _runtime‚23080TŒ#•5˜P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1000443020806144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16207214327808 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13233572161652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3657466526050304 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3656024346479856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3670722292236800 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1000443388897280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚23080.942935423 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13254420750336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3657466526050304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110289748 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228237219 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33842848 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1016650603225088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110289754 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117937401 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3670720920627200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117937401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16207163060736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72566466 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1016650183866880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.currenÍÆçt‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3670722292236800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3670720920627200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72566832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110289748 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3657466499876864´—Š–#(–5P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228237219 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33842848 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1016650603225088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110289754 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117937401 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3670720920627200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117937401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16207163060736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72566466 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1016650183866880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3670722292236800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3670720920627200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72566832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚366 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110289748 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3657466499876864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1000443020806144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16207214327808 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13233572161652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3657466526050304 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3656024346479856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚361 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3670722292236800 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1000443388897280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚23080.942935423 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13254420750336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3657466526050304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110289748 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006¢Z:™è4—5/ _wandbruntime‚23082  _runtime‚23082ã{¾''˜5¡N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33845488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16208468083712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117946483 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1000521607172096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3671002224042496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13256788701696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 trainer/global_step‚12699 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117956561 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1000521975263232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228244598 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13256788701696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3657745409167360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3656303173681600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33845483 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3671002224042496 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1016730011412992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13234592963404 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13233248352006 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228254682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3657745435340800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13255443258368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110298115 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 % _timestamp‚1.7848623190568492e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110298121 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117946483 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16208404240896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3669536396605638 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _step‚1059 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228244598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38726829 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3657745435340800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110298115 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3671000852425728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38726464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3669537766645004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72571947 0ÃC,'™5¦N D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _step‚1059 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228244598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38726829 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3657745435340800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110298115 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3671000852425728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38726464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3669537766645004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72571947 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33845488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16208468083712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117946483 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1000521607172096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3671002224042496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13256788701696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 trainer/global_step‚12699 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117956561 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1000521975263232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228244598 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13256788701696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3657745409167360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3656303173681600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33845483 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3671002224042496 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1016730011412992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13234592963404 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13233248352006 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228254682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3657745435340800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13255443258368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110298115 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 % _timestamp‚1.7848623190568492e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110298121 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117946483 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16208404240896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3669536396605638 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118393353 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118403417 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020642346784256 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1004372707194880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33974848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚367 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13284676044460 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16269584128000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3684717574708224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110708104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3684718946317824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚23167.437381821 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118393353 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38868535 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3671411990575104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72843378 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3683249136641084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118403417 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 % _timestamp‚1.7848624040782626e+09 _step‚1062 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224ÄoW&Õ#®5P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118393353 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38868535 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3671411990575104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72843378 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55461376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3683249136641084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118403417 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 % _timestamp‚1.7848624040782626e+09 _step‚1062 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33974843 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3671411964401664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3669964460596624 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3669964435168656 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13306955742720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1020641923231744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16269639589376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3683247766601830 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72843750 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3684718946317824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 trainer/global_step‚12749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229101451 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118393353 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118403417 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020642346784256 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1004372707194880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33974848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚367 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13284676044460 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16269584128000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3684717574708224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110708104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3684718946317824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚23167.437381821¦r‹¤ß4¯5/ _wandbruntime‚23168  _runtime‚23168ŸÇ7•('°5¢N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118402435 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33977483 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110716471 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110716465 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13306632814592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229118900 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3684997506506752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1020722177707520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13306632814592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 trainer/global_step‚12749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229128984 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1004451293560832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229118900 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _runtime‚23168.915077058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13307978257920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110716471 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118412513 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3671690873692160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72849269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72848899 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16270884146688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16270822401024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3683527589177246 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38871781 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3684998878123520 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3683528959216612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33977488 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1020721747870720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 % _timestamp‚1.7848624055574403e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3684998878123520 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3671690899865600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38871416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3670243236942432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400úGjZ-'±5§N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229128984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13285696846212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13307978257920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3684998878123520 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3683528959216612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33977488 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1020721747870720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 % _timestamp‚1.7848624055574403e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3684998878123520 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3671690899865600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38871416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3670243236942432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118402435 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33977483 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110716471 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110716465 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13306632814592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229118900 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3684997506506752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚365 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1020722177707520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13306632814592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 trainer/global_step‚12749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229128984 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1004451293560832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229118900 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _runtime‚23168.915077058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13307978257920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110716471 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118412513 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3671690873692160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72849269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72848899 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16270884146688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16270822401024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3683527589177246 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38871781 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 Q t‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd2118.in‚0 network.recv‚ 1.652552e+06 disk.dm-0.in‚ 119.03125& disk.nvme0n1.out‚57016.06982421875 disk.drbd1550.out‚0( disk./.usagePercent‚70.51417722114391ó¹¿5‚(:’ –®‹Ó¢Öé³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚252.443' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.14766666666668 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ïÕ¨LÀ5‚(:Á  –®‹Ó˜‚´ disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 910.24609375 disk.drbd1673.in‚0 disk.sda2.out‚ 910.24609375 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.756149927775066) disk./.usagePercent‚70.51417508776548 network.sent‚ 2.22196e+07 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 119.03125 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0) proc.memory.availableMB‚ 1943.37890625% disk./.usageGB‚1260.8650665283203 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 network.recv‚ 1.653532e+06 disk.drbd1552.out‚0 disk.drbd1285.in‚0 cpu‚25.33989991740275 disk.drbd1550.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1938.out‚0# disk.dm-0.out‚57016.42529296875 disk.drbd2118.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1938.in‚0# proc.memory.rssMB‚ 2889.19140625 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 115.853515625 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.dm-0.in‚ 119.03125 disk.drbd1828.in‚0 disk.drbd1931.out‚0$ memory_percent‚92.09237098693848 disk.drbd2041.in‚0& disk.nvme0n1.out‚57016.42529296875 disk.drbd1226.in‚0 disk.sda1.in‚0€Í4Á5/ _wandbruntime‚23254  _runtime‚23254¶¬‹@ÊÂ5Ä & lr-AdamW/pg1‚2.200611121162625e-06 & lr-AdamW/pg2‚2.200611121162625e-06 trainer/global_step‚12799 % _timestamp‚1.7848624911909077e+09 _runtime‚23254.548151018 _step‚1065ÆÅÏÃ5É & lr-AdamW/pg1‚2.200611121162625e-06 & lr-AdamW/pg2‚2.200611121162625e-06 trainer/global_step‚12799 % _timestamp‚1.7848624911909077e+09 _runtime‚23254.548151018 _step‚1065©@&§™4Ä5/ _wandbruntime‚23254  _runtime‚23254la(Å5˜P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3685357455099904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13356799862784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3698714228789248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111126454 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13335779927268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3697238959173438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13358145298944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73121383 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1008302025492480 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3698714228789248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111126454 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13335779927268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3697238959173438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13358145298944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73121383 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1008302025492480 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3698714228789248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚369 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3685636338216960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _step‚1067 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111134821 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230003286 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3698994160587776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229993202 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73126552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3697518781748854 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118858387 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34109488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13336800729020 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13357822370816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 trainer/global_step‚12799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111134815 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚355 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1008380243767296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚23256.029460216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13359167814144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3698995532204544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3684183325631232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OADeviceStatsMonitor.on_train_batch_end/aclõòWátive.small_pool.allocated‚ 118868465 % _timestamp‚1.7848624926718073e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229993202 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16333242844672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39017424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39017069 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398Äf‚ç-'É5§N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13335456117622 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3685636338216960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13359167814144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73126912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230003286 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3698994160587776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229993202 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73126552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3697518781748854 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118858387 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34109488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13336800729020 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13357822370816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 trainer/global_step‚12799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111134815 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚355 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1008380243767296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚23256.029460216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13359167814144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3698995532204544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3684183325631232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118868465 % _timestamp‚1.7848624926718073e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229993202 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16333242844672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39017424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39017069 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3698995532204544 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3685636338216960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _step‚1067 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111134821 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1012230975698944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16394417697792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119315321 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13409334855168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3712712254479872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39158889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34238843 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3699302893451264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 % _timestamp‚1.7848625778790572e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34238848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3699302919624704 XEDeviceStatsMonitor.on_train_batch_start/acti¦üo… ve_bytes.small_pool.freed‚13407989419008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3711231521784300 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111544804 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3712710882870272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230860125 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1012231343790080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16394479450624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1012230975698944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16394417697792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119315321 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13409334855168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3712712254479872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39158889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34238843 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3699302893451264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 % _timestamp‚1.7848625778790572e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34238848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3699302919624704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13407989419008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3711231521784300 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111544804 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3712710882870272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230860125 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1012231343790080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16394479450624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73402944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230877588 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1012309930156032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3698123439748000 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028705652119040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13409011927040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119324417 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73403301 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13410357370368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39161461 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3698123414320032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13409011927040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39161813 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3712992186285568 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3699581802741760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111553171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3711511344359828 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3712990814668800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13410357370368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13386560000430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111553165 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34241488 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73402944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230877588 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1012309930156032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3698123439748000 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028705652119040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13409011927040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119324417 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73403301 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13410357370368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39161461 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3698123414320032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13409011927040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39161813 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3712992186285568 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3699581802741760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111553171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3711511344359828 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3712990814668800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13410357370368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13386560000430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111553165 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34241488 F&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30945172905921936 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1768982708454132 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05336398631334305 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36406010389328003 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6563348770141602 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3118620216846466 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35829466581344604 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.729937732219696 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_4‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6625420451164246 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 *jet/accuracy_1_of_2‚0.7916690707206726 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5276833772659302 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 )loss/val_total_loss‚7.722965915696001 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9239448308944702 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027164917439222336 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7206024527549744 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7600889801979065 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.715584933757782 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784862650667761e+09 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6429777145385742 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6993283629417419 +jet/accuracy_1_of_1‚0.37365004420280457 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03240741416811943 (particle/accuracy‚0.8552071452140808 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6657652854919434 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.300245076417923 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7056443095207214 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8077439665794373 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18902622163295746 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 particle/accuracy_1_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3910154402256012 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29703840613365173 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.641387939453125 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 *jet/accuracy_1_of_4‚0.9832039475440979 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5175161957740784 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3697248101234436 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 _runtime‚23414.026033598 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7813112139701843 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.228875532746315 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17307327687740326 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2707745134830475 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27679118514060974 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.15368707478046417 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.553892970085144 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.694459080696106 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37231841683387756 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35829466581344604 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.18401774764060974 *jet/accuracy_1_of_3‚0.9387298822402954 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.044089529663324356 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3871796131134033 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8421576619148254¬‡gŸ ï5v 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6993283629417419 +jet/accuracy_1_of_1‚0.37365004420280457 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03240741416811943 (particle/accuracy‚0.8552071452140808 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6657652854919434 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.300245076417923 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7056443095207214 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8077439665794373 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18902622163295746 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 particle/accuracy_1_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3910154402256012 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29703840613365173 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.641387939453125 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 *jet/accuracy_1_of_4‚0.9832039475440979 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5175161957740784 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3697248101234436 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 _runtime‚23414.026033598 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7813112139701843 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.228875532746315 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17307327687740326 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7641026377677917 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2707745134830475 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27679118514060974 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.15368707478046417 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.553892970085144 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.694459080696106 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37231841683387756 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35829466581344604 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.18401774764060974 *jet/accuracy_1_of_3‚0.9387298822402954 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.044089529663324356 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3871796131134033 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8421576619148254 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7760825753211975 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6608628034591675 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16067703068256378 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.690180778503418 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6336421966552734 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6353898048400879 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2816733717918396 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚45 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.27165696024894714 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6391066312789917 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4079034626483917 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3498413562774658 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4590897262096405 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7120177149772644 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 +particle/sensitivity‚0.6431614756584167 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5870780348777771 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7221772074699402 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7279393076896667 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7998269200325012 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.883858859539032 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚…RÝYó0.3589112162590027 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.37048348784446716 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7992503046989441 7"Purity/0FRt*SRqqt1FBt/event_purity‚0.87553870677948 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9993678331375122 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 trainer/global_step‚12879 ,jet/accuracy_4_of_4‚0.018756305798888206 'particle/f_score‚0.6558651924133301 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.679616391658783 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7"Purity/2FRt*SRqqt0FBt/event_purity‚0.07623241096735 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07885482162237167 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6675848364830017 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26751476526260376 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _step‚1073 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6428624987602234 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6950875520706177 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8019219040870667 particle/accuracy_1_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2901070713996887 6validation_average_jet_accuracy‚0.4075183570384979 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3844899535179138 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 +jet/accuracy_2_of_2‚0.17364704608917236 +jet/accuracy_3_of_4‚0.10971427708864212 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09235002100467682 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.034647662192583084 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7583100199699402 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 *jet/accuracy_2_of_3‚0.6043210625648499 +particle/specificity‚0.9131274819374084 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028776420280337334 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6468684077262878 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7206576466560364 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6134615540504456 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3568480908870697 /particle/accuracy_4_of_4‚0.9671996831893921 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38946348428726196 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6934504508972168 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2253507375717163 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7704960703849792 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6468684077262878 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30945172905921936 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1768982708454132 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05336398631334305 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36406010389328003 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6563348770141602 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3118620216846466 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35829466581344604 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.729937732219696 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_4‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6625420451164246 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 *jet/accuracy_1_of_2‚0.7916690707206726 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5276833772659302 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 )loss/val_total_loss‚7.722965915696001 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9239448308944702 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027164917439222336 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7206024527549744 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7600889801979065 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.715584933757782 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784862650667761e+09 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6429777145385742 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325±&‡_ð5‚(:’ »¯‹Óÿ´·³ gpu.0.gpu‚89.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚289.486' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.49533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú7QâLñ5‚(:Á  »¯‹Ó‘„Ö³ disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda.in‚ 0.0390625 disk.drbd1226.out‚0 disk.drbd1584.in‚0 cpu‚25.341464387251033 disk.drbd1673.out‚0 disk.drbd1289.out‚0" disk.dm-0.out‚57020.9033203125 disk.drbd1510.out‚0 network.sent‚ 2.2539236e+07 disk.sda2.in‚ 0.0390625 disk.drbd1938.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1584.out‚0( proc.memory.availableMB‚ 1930.1328125 disk.sda1.in‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.809031168619793 network.recv‚ 1.665755e+06 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1260.8618545532227 disk.drbd1828.out‚0 disk.drbd1285.in‚0$ memory_percent‚92.14626948038736 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 123.6328125 disk.rbd1.out‚116.7939453125 disk.drbd1817.in‚0 disk.sda.out‚ 914.91796875 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 123.6328125 proc.memory.rssMB‚ 2902.1875% disk.nvme0n1.out‚57020.9033203125( disk./.usagePercent‚70.5139954573035 disk.sda2.out‚ 914.91796875 disk.drbd2078.in‚0 disk.drbd1356.out‚0nDÅMò5‚(:  ʯ‹Ó½õâ³ disk.dm-0.in‚ 123.6328125 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚116.7939453125# disk.dm-0.out‚57021.71240234375 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625) proc.memory.availableMB‚ 1930.12890625 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 cpu‚25.3414270098054 disk.sda.out‚ 915.22265625 disk.drbd2078.in‚0 disk.drbd1510.in‚0 proc.memory.rssMB‚ 2902.1875 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚57021.71240234375 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 network.sent‚ 2.2593213e+07 disk.drbd1552.in‚0$ memory_percent‚92.14628537495932 network.recv‚ 1.667599e+06$ disk./.usageGB‚1260.861888885498 disk.sda1.in‚0 disk.sda2.out‚ 915.22265625 proc.cpu.threads‚17 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.51399737734405 disk.drbd2111.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.809031168619793 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 123.6328125 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0ÿKÑžó5‚(:“ ʯ‹ÓºÀ³ gpu.0.gpu‚42.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚314.437' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.81233333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÑ˜w’ô5‚(:‡ Ù¯‹Ó֩س gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚298.896' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ÞíåMõ5‚(:  Ù¯‹ÓôÂù³ disk.rbd1.out‚ 116.826171875 disk.sda1.out‚0 disk.drbd1965.in‚0$ memory_percent‚92.14628537495932) proc.memory.availableMB‚ 1930.12890625 proc.memory.rssMB‚ 2902.1875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 cpu‚25.34136818865703 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0# disk.dm-0.out‚57022.58154296875 network.sent‚ 2.2596041e+07 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 915.67578125 disk.sda.in‚ 0.0390625 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0) disk./.usagePercent‚70.51399737734405 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 123.6328125$ disk./.usageGB‚1260.861888885498 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1289.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚57022.58154296875 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 123.6328125 disk.drbd1817.out‚0 network.recv‚ 1.667731e+06 disk.sda.out‚ 915.67578125 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.809031168619793 disk.drbd1552.in‚0 disk.drbd1356.out‚0:MÔí4ö5/ _wandbruntime‚23447  _runtime‚23447´½O[Ì÷5Æ ' lr-AdamW/pg1‚1.8512678175414785e-06 ' lr-AdamW/pg2‚1.8512678175414785e-06 trainer/global_step‚12899 % _timestamp‚1.7848626840375304e+09 _runtime‚23447.394814175 _step‚1074Pû ûÑø5Ë _step‚1074 ' lr-AdamW/pg1‚1.8512678175414785e-06 ' lr-AdamW/pg2‚1.8512678175414785e-06 trainer/global_step‚12899 % _timestamp‚1.7848626840375304e+09 _runtime‚23447.394814175²Ý½4ù5/ _wandbruntime‚23447  _runtime‚23447¯fÿ(ú5˜P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112052580 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3715809060874240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119818580 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73711558 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34377740 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39333823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3714343702386224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3715809060874240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13464797411840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3727785954307131 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3715809034700800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231861096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112052586 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3727787324346385 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13442251920907 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 % _timestamp‚1.7848626840386627e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73711859 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1016209293509632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13443596532193 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13466142848000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112052580 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13464797411840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231871166 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3729275203722240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3714343727814192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3729273832112640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1016209661600768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34377735 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39334119 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1032673718993920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3729275203722240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119818580 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224³auÜû5P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112052586 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112052580 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3715809060874240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119818580 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73711558 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34377740 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39333823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3714343702386224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3715809060874240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13464797411840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3727785954307131 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3715809034700800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231861096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112052586 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3727787324346385 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13442251920907 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 % _timestamp‚1.7848626840386627e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73711859 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1016209293509632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13443596532193 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13466142848000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112052580 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13464797411840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231871166 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3729275203722240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3714343727814192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3729273832112640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1016209661600768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34377735 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39334119 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1032673718993920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3729275203722240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119818580 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚40774144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3729555135527936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119827676 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _step‚1076 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1032753174941184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112060947 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119827676 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3716087943991296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3728067146921913 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3714622529587968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3716087970164736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231888629 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231878545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16465295065600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3714622504160000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13444617333945 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13467165363200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112060953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13443272722547 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231888629 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3716087943991296 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112060953 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34380375 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73717509 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13465819919872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3729553763911168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39337129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39336833 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73717208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13467165363200 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119817598 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1032753583806464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚40774144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3729555135527936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119827676 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 _step‚1076 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1032753174941184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112060947 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119827676 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3716087943991296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3728067146921913 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3714622529587968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3716087970164736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231888629 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231878545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16465295065600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3714622504160000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13444617333945 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13467165363200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112060953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13443272722547 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231888629 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3716087943991296 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112060953 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34380375 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73717509 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13465819919872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3729553763911168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39337129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39336833 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73717208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13467165363200 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119817598 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1032753583806464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34509740 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112470930 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13517332404224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13515986968064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16526831430656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚339 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3741777146878739 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73992376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3743271857803264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13494700415001 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112470936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3729754499225600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120274532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120264468 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3741778516917993 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 _step‚1079 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232745468 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16526882697728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120264468 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 trainer/global_step‚12949 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13493355803715 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _runtime‚23533.865922938 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13515986968064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 % _timestamp‚1.7848627705053005e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3743270486193664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39482641 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34509735 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232735398 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3743270486193664 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1036665862596096 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13517332404224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1020138611807232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3728283816502992 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3743270486193664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39482641 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34509735 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232735398 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3743270486193664 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1036665862596096 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13517332404224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1020138611807232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3728283816502992 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34509740 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112470930 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13517332404224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13515986968064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16526831430656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚339 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3741777146878739 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73992376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3743271857803264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13494700415001 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112470936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3729754499225600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120274532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120264468 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3741778516917993 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 _step‚1079 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232745468 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16526882697728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120264468 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 trainer/global_step‚12949 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13493355803715 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 _runtime‚23533.865922938 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13515986968064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 % _timestamp‚1.7848627705053005e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34512380 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚347 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3728562592848800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13517009476096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16528079514624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13518354919424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3743550417992192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16528132871680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13494376605355 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73997962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3743551789608960 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120273550 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112479303 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3728562618276768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34512375 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1036745699136000 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232752847 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3730033434689536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13518354919424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚12949 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3743551789608960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 _runtime‚23535.337777484 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120283628 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232762931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120283628 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _step‚1080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13517009476096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3742058339493521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13495721216753 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚342 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1036745277687808 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3742056969454155 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3730033408516096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segŸÞ®ýÝments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120273550 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39485587 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1020217198173184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112479303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112479297 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264Aˆ”-'–6§N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112479303 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3728562618276768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34512375 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1036745699136000 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232752847 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3730033434689536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13518354919424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚12949 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3743551789608960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 _runtime‚23535.337777484 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120283628 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232762931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120283628 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _step‚1080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13517009476096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3742058339493521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13495721216753 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚342 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1036745277687808 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3742056969454155 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3730033408516096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120273550 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39485587 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1020217198173184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112479303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112479297 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73998309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39485929 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34512380 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚347 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3728562592848800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13517009476096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16528079514624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13518354919424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3743550417992192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16528132871680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13494376605355 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73997962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3743551789608960 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120273550 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400¸ÔÞ¶4—6/ _wandbruntime‚23535  _runtime‚23535è?"¨˜6‰ 3loss/SRqqt2/assignment_loss‚0.43232028835299346 2loss/FBt1/assignment_loss‚0.009515254148111982 1loss/SRqqt1/detection_loss‚0.8763956735183068 /loss/FBt1/detection_loss‚0.4544185273397457 %loss/total_loss‚7.419878081135312 epoch‚46 _runtime‚23535.373998871 2loss/FBt2/assignment_loss‚0.008547172683411576 /loss/FRt1/detection_loss‚0.5475930344372824 /loss/FRt2/detection_loss‚1.3302653840845786 0loss/FBt2/detection_loss‚0.42643577837690794 % _timestamp‚1.7848627720167925e+09 0loss/FRt1/assignment_loss‚1.5903897209736564 0loss/FRt2/assignment_loss‚0.6781950766170347 2loss/SRqqt1/assignment_loss‚0.4881463604711779 1loss/SRqqt2/detection_loss‚0.5776558101321052 trainer/global_step‚12949 _step‚1081.]Tû™6Ž 2loss/SRqqt1/assignment_loss‚0.4881463604711779 1loss/SRqqt2/detection_loss‚0.5776558101321052 trainer/global_step‚12949 _step‚1081 3loss/SRqqt2/assignment_loss‚0.43232028835299346 2loss/FBt1/assignment_loss‚0.009515254148111982 1loss/SRqqt1/detection_loss‚0.8763956735183068 /loss/FBt1/detection_loss‚0.4544185273397457 %loss/total_loss‚7.419878081135312 epoch‚46 _runtime‚23535.373998871 2loss/FBt2/assignment_loss‚0.008547172683411576 /loss/FRt1/detection_loss‚0.5475930344372824 /loss/FRt2/detection_loss‚1.3302653840845786 0loss/FBt2/detection_loss‚0.42643577837690794 % _timestamp‚1.7848627720167925e+09 0loss/FRt1/assignment_loss‚1.5903897209736564 0loss/FRt2/assignment_loss‚0.6781950766170347¹û‹c/š6‚(:’ °‹Óɣݳ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚297.113' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.03766666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡€×!P›6‚(:Å  °‹Ó’Êý³ disk.drbd2078.out‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0 disk.rbd1.out‚117.2724609375 disk.drbd1510.out‚0 network.recv‚ 1.674417e+06 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 network.sent‚ 2.2717219e+07 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 123.63671875 disk.sda1.in‚0 disk.sda.in‚ 0.0390625 disk.sda.out‚ 918.4296875 disk.drbd1673.out‚0* proc.memory.percent‚11.809031168619793& disk.nvme0n1.out‚57025.43408203125 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1289.in‚0% disk./.usageGB‚1260.8624114990234 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 123.63671875 disk.sda2.out‚ 918.4296875 disk.drbd1938.in‚0 cpu‚25.341077928425303 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd1356.in‚0# disk.dm-0.out‚57025.43408203125 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0) proc.memory.availableMB‚ 1929.51171875$ memory_percent‚92.14879671732584 proc.memory.rssMB‚ 2902.1875) disk./.usagePercent‚70.51402660462826 disk.drbd1965.in‚0Q¼.ÿœ6‚(:’ Ñ°‹ÓÓ¶¯³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.85' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.28333333333335 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð(sFL6‚(:Á  Ñ°‹Ó¤ŸÍ³ disk.rbd1.out‚ 117.3046875 disk.drbd1673.in‚0$ memory_percent‚92.14879671732584* proc.memory.percent‚11.809031168619793 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 918.83984375 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 network.recv‚ 1.674525e+06 disk.drbd2111.in‚0 disk.drbd1584.out‚0 proc.memory.rssMB‚ 2902.1875 disk.dm-0.in‚ 123.63671875 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.sda.in‚ 0.0390625) disk./.usagePercent‚70.51402340456063 disk.drbd1938.out‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 918.83984375" disk.dm-0.out‚57026.3623046875 disk.drbd2111.out‚0 disk.drbd2118.out‚0 network.sent‚ 2.2717327e+07 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚57026.3623046875 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 cpu‚25.34103150120107% disk./.usageGB‚1260.8623542785645 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0) proc.memory.availableMB‚ 1929.51171875 disk.sda1.in‚0tÊbž6‚(:’ à°‹ÓŠÅ·³ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚290.725' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.90833333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·jLŸ6‚(:Á  à°‹Ó“Ù³ disk.nvme0n1.in‚ 123.63671875 disk.sda.out‚ 919.05078125 disk.drbd1673.in‚0 network.sent‚ 2.2717815e+07 disk.drbd1356.out‚0 disk.drbd1226.in‚0) proc.memory.availableMB‚ 1929.51171875 proc.memory.rssMB‚ 2902.1875 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0 cpu‚25.34100520555153 disk.sda1.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.8623733520508% disk.nvme0n1.out‚57026.5107421875 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.sda.in‚ 0.0390625" disk.dm-0.out‚57026.5107421875 disk.drbd2041.out‚0 disk.drbd1828.in‚0 network.recv‚ 1.675546e+06 disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚ 0.0390625 disk.dm-0.in‚ 123.63671875 disk.drbd1226.out‚0$ memory_percent‚92.14879671732584 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.809031168619793) disk./.usagePercent‚70.51402447124984 disk.drbd1673.out‚0 disk.sda2.out‚ 919.05078125 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 117.3046875FnmÞ’ 6‚(:‡ ï°‹ÓžŽÛ³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.529' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚99.843 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÐG¡6‚(:¼  ï°‹Ó¿¶ý³* proc.memory.percent‚11.809031168619793 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1510.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 919.4765625 disk.drbd1938.out‚0# disk.dm-0.out‚57026.94873046875 proc.memory.rssMB‚ 2902.1875 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0) disk./.usagePercent‚70.51402340456063 disk.drbd1226.in‚0 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1931.out‚0 disk.drbd1938.in‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 123.63671875 disk.rbd1.out‚ 117.46875 disk.drbd1584.in‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.in‚0 cpu‚25.34095738030261 disk.drbd1285.in‚0 disk.sda.out‚ 919.4765625 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.0390625 disk.drbd2078.in‚0$ memory_percent‚92.14363098144531 network.sent‚ 2.2717923e+07 disk.drbd1828.out‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 1930.78125 disk.drbd1828.in‚0& disk.nvme0n1.out‚57026.94873046875 disk.drbd1550.out‚0 network.recv‚ 1.675654e+06% disk./.usageGB‚1260.8623542785645 disk.nvme0n1.in‚ 123.63671875 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0žÝ¸$¢6‚(:’ þ°‹ÓŒ¡é³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.122' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.70733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¨‚L£6‚(:Á  þ°‹ÓÍ܆´) disk./.usagePercent‚70.51402340456063 disk.drbd1550.out‚0 disk.sda2.in‚ 0.0390625$ memory_percent‚92.14463233947754) proc.memory.availableMB‚ 1930.53515625 disk.drbd1226.out‚0# disk.dm-0.out‚57027.49169921875 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 919.69140625 proc.cpu.threads‚17 network.sent‚ 2.2717989e+07 disk.drbd2041.out‚0 disk.rbd1.out‚ 117.46875 disk.nvme0n1.in‚ 123.63671875 disk.drbd1817.out‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.809031168619793 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 proc.memory.rssMB‚ 2902.1875 cpu‚25.340899511783263 disk.drbd1356.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0& disk.nvme0n1.out‚57027.49169921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0390625 disk.drbd1828.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 919.69140625 network.recv‚ 1.67572e+06% disk./.usageGB‚1260.8623542785645 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd2111.in‚0 disk.drbd2118.out‚0boäŠ4¤6/ _wandbruntime‚23620  _runtime‚23620NôÊ¥6Ä & lr-AdamW/pg1‚1.531658767056665e-06 & lr-AdamW/pg2‚1.531658767056665e-06 trainer/global_step‚12999 % _timestamp‚1.7848628573947372e+09 _runtime‚23620.752907161 _step‚10824ZíϦ6É & lr-AdamW/pg2‚1.531658767056665e-06 trainer/global_step‚12999 % _timestamp‚1.7848628573947372e+09 _runtime‚23620.752907161 _step‚1082 & lr-AdamW/pg1‚1.531658767056665e-06º–}Ö4§6/ _wandbruntime‚23620  _runtime‚236203ª(ä‘!¨6˜P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3757267140274688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16589325120512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233619770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024068298195968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3757267140274688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3755768339450347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3757268511884288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120730484 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34641735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1040657676680704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3742223879763824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 % _timestamp‚1.7848628573955357e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 _step‚1083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39630257 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1024067930104832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/s°å2egment.small_pool.peak‚825 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3743699989923840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74271997 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚12999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3743699963750400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13568521960448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16589378484736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233609700 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74271630 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¢>ã7#(©6P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120720420 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _runtime‚23620.753017611 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112889280 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 _step‚1083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39630257 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1024067930104832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3743699989923840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74271997 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚12999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3743699963750400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13568521960448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16589378484736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233609700 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74271630 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3757267140274688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16589325120512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233619770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024068298195968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3757267140274688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3755768339450347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3757268511884288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120730484 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34641735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1040657676680704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3742223879763824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 % _timestamp‚1.7848628573955357e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233637233 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120729502 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74277540 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚371 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3756048162025763 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120739580 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3756049532065129 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12999 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3743978873040896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233637233 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3743978899214336 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3757547072073216 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3743978899214336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3742502681537600 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13569544475648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3743978873040896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112897647 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120729502 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39633160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13545480488163 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74277169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16590619636224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3757548443689984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 _runtime‚23622.226174655 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024146884561920 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024146516470784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120739580 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13546825099561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233627149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39632794 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233627149 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34644380 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0NØê-'¬6§N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 _runtime‚23622.226174655 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024146884561920 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024146516470784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120739580 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13546825099561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233627149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39632794 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233627149 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34644380 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34644375 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233637233 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120729502 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74277540 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚371 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3756048162025763 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120739580 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3756049532065129 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12999 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3743978873040896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233637233 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3743978899214336 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3757547072073216 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3743978899214336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3742502681537600 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13569544475648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3743978873040896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112897647 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120729502 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39633160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13545480488163 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74277169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16590619636224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3757548443689984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¼}R­4­6/ _wandbruntime‚23622  _runtime‚23622¾êÁ¦b®6ˆ 0loss/FRt1/assignment_loss‚1.5332764468765616 2loss/SRqqt2/assignment_loss‚0.5041170409973393 1loss/SRqqt2/detection_loss‚0.6122457775678214 trainer/global_step‚12999 % _timestamp‚1.7848628589093432e+09 1loss/SRqqt1/detection_loss‚0.7856051559642363 /loss/FBt2/detection_loss‚0.4191743121846886 epoch‚46 0loss/FRt2/assignment_loss‚0.7348075390145218 3loss/SRqqt1/assignment_loss‚0.49545789001120927 3loss/FBt1/assignment_loss‚8.417899988370828e-05 2loss/FBt2/assignment_loss‚0.010331625531421708 /loss/FRt1/detection_loss‚0.5316622708972542 /loss/F 3½ë¬Bt1/detection_loss‚0.6041750360183932 _runtime‚23622.266525942 _step‚1085 .loss/FRt2/detection_loss‚1.471362682147894 %loss/total_loss‚7.702299956211226øÑ£(¯6 0loss/FRt1/assignment_loss‚1.5332764468765616 2loss/SRqqt2/assignment_loss‚0.5041170409973393 1loss/SRqqt2/detection_loss‚0.6122457775678214 trainer/global_step‚12999 % _timestamp‚1.7848628589093432e+09 1loss/SRqqt1/detection_loss‚0.7856051559642363 /loss/FBt2/detection_loss‚0.4191743121846886 epoch‚46 0loss/FRt2/assignment_loss‚0.7348075390145218 3loss/SRqqt1/assignment_loss‚0.49545789001120927 3loss/FBt1/assignment_loss‚8.417899988370828e-05 2loss/FBt2/assignment_loss‚0.010331625531421708 /loss/FRt1/detection_loss‚0.5316622708972542 /loss/FBt1/detection_loss‚0.6041750360183932 _runtime‚23622.266525942 _step‚1085 .loss/FRt2/detection_loss‚1.471362682147894 %loss/total_loss‚7.702299956211226½ÓÏAhž°6‚(:“ ±‹ÓûÙÙ³ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.219' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.40633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÃRQ±6‚(:Æ  ±‹ÓÜü³ disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0& disk.nvme0n1.out‚57027.90771484375 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1931.out‚0 disk.drbd1550.in‚0 proc.memory.rssMB‚ 2902.1875* proc.memory.percent‚11.809031168619793 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 network.sent‚ 2.2718097e+07 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0390625$ memory_percent‚92.14488665262857 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚117.5009765625 cpu‚25.34085261609151) proc.memory.availableMB‚ 1930.47265625) disk./.usagePercent‚70.51402340456063 disk.drbd1817.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.0390625 disk.dm-0.in‚ 123.63671875 disk.drbd1552.in‚0 disk.sda.out‚ 920.31640625 disk.drbd1673.in‚0 disk.sda2.out‚ 920.31640625 disk.drbd1356.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8623542785645 disk.drbd1289.in‚0# disk.dm-0.out‚57027.90771484375 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 network.recv‚ 1.675828e+06 disk.drbd1673.out‚0 disk.nvme0n1.in‚ 123.63671875Üû(äP²6‚(:Å  œ±‹Óã Ø³ disk.drbd1817.out‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 920.4921875 disk.sda2.out‚ 920.4921875 disk.drbd2078.out‚0 disk.rbd1.out‚117.5009765625 disk.drbd1285.out‚0 network.recv‚ 1.676804e+06 disk.drbd1931.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚57028.23388671875 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.14488665262857 proc.cpu.threads‚17% disk./.usageGB‚1260.8623542785645 disk.drbd2118.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd2041.in‚0 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0) disk./.usagePercent‚70.51402340456063 disk.drbd1965.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.809031168619793) proc.memory.availableMB‚ 1930.47265625 disk.drbd1552.out‚0 disk.drbd1289.out‚0 proc.memory.rssMB‚ 2902.1875 disk.sda1.in‚0 disk.drbd1226.out‚0 network.sent‚ 2.2718585e+07 disk.drbd1289.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1552.in‚0 disk.drbd1931.out‚0# disk.dm-0.out‚57028.23388671875 disk.drbd1510.in‚0 disk.sda.in‚ 0.0390625 disk.nvme0n1.in‚ 123.63671875 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚ 0.0390625 disk.drbd1356.out‚0 cpu‚25.340816814083624ª¸Gp›³6‚(: œ±‹ÓÝå¹³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚60.0 gpu.0.powerWatts‚107.992' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.99733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0힣ž´6‚(:“ «±‹Ó‚ûå³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚307.219' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.40633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K„0.Eµ6‚(:º  «±‹ÓµÚ‰´ network.sent‚ 2.2718651e+07 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd1510.out‚0) proc.memory.availableMB‚ 1930.47265625 disk.drbd1550.in‚0$ memory_percent‚92.14488665262857* proc.memory.percent‚11.809031168619793 disk.drbd2078.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1552.out‚0 disk.drbd1550.out‚0% disk./.usageGB‚1260.8633346557617 disk.sda1.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 920.90625 disk.drbd1289.out‚0 disk.dm-0.out‚57029.23828125 disk.drbd2078.out‚0 disk.rbd1.out‚117.8369140625 disk.drbd1510.in‚0 disk.sda2.out‚ 920.90625 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1356.in‚0 cpu‚25.340779548074508 disk.drbd2118.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda.in‚ 0.0390625 disk.drbd1285.in‚0# disk.nvme0n1.out‚57029.23828125 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 123.63671875 proc.memory.rssMB‚ 2902.1875 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1584.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1285.out‚0 network.recv‚ 1.67687e+06 disk.drbd1226.out‚0) disk./.usagePercent‚70.51407823238574 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0Üè¢iP¶6‚(:Å  º±‹Ó½©Í³ network.sent‚ 2.2718717e+07 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚ 0.0390625 disk.sda1.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1673.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚57029.4306640625 network.recv‚ 1.676936e+06 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.dm-0.in‚ 123.63671875 disk.rbd1.out‚117.8369140625 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1817.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 921.42578125 disk.drbd1356.in‚0$ memory_percent‚92.14491844177246 proc.memory.rssMB‚ 2902.1875 cpu‚25.340744252904038" disk.dm-0.out‚57029.4306640625 disk.drbd1552.out‚0% disk./.usageGB‚1260.8633346557617 disk.drbd1673.in‚0 disk.drbd1828.in‚0* proc.memory.percent‚11.809031168619793 disk.sda.out‚ 921.42578125 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0) proc.memory.availableMB‚ 1930.46484375) disk./.usagePercent‚70.51407823238574 disk.drbd1931.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 123.63671875¨JÒ›·6‚(: º±‹Ó©Ç­³ gpu.0.gpu‚95.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚252.1' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.03333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›f˜Ð›¸6‚(: ɱ‹Ó˕ȳ gpu.0.gpu‚8.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚108.082' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.02733333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏCìK¹6‚(:À  ɱ‹ÓÉÖç³ network.sent‚ 2.2718825e+07 disk.drbd1931.out‚0 disk.dm-0.in‚ 123.63671875 disk.sda.in‚ 0.0390625 disk.drbd1673.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1289.in‚0 disk.drbd2118.out‚0* proc.memory.percent‚11.809031168619793 cpu‚25.340696938921944 disk.drbd1289.out‚0) proc.memory.availableMB‚ 1930.46484375 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 922.2265625 disk.drbd1673.in‚0 disk.sda2.out‚ 922.2265625 disk.drbd1938.out‚0! disk.dm-0.out‚57030.599609375$ disk.nvme0n1.out‚57030.599609375 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd2118.in‚0 disk.rbd1.out‚ 117.869140625 network.recv‚ 1.677044e+06 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1817.in‚0$ memory_percent‚92.14491844177246 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.nvme0n1.in‚ 123.63671875 proc.memory.rssMB‚ 2902.1875% disk./.usageGB‚1260.8633346557617 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.51407823238574 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1817.out‚0j³]º6‚(:’ ر‹ÓèŸÚ³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚298.984' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.66133333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÚ‹ÃJ»6‚(:¿  ر‹ÓþÔù³ disk.drbd1817.out‚0 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 123.63671875 proc.cpu.threads‚17 network.sent‚ 2.277421e+07 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 117.869140625 disk.drbd1552.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 cpu‚25.340660374926014 disk.sda1.out‚0 disk.sda2.in‚ 0.0390625! disk.dm-0.out‚57030.759765625 disk.drbd2111.out‚0 disk.drbd2041.out‚0 disk.drbd1510.in‚0) disk./.usagePercent‚70.51407823238574 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1930.46484375 disk.drbd1285.out‚0$ memory_percent‚92.14491844177246 disk.drbd1938.in‚0% disk./.usageGB‚1260.8633346557617 disk.drbd1817.in‚0$ disk.nvme0n1.out‚57030.759765625 disk.dm-0.in‚ 123.63671875 disk.drbd1289.out‚0 proc.memory.rssMB‚ 2902.1875 disk.sda.out‚ 922.4921875 disk.drbd1673.in‚0 disk.sda2.out‚ 922.4921875 disk.drbd1828.in‚0 network.recv‚ 1.679795e+06 disk.sda.in‚ 0.0390625 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0* proc.memory.percent‚11.809031168619793 disk.drbd1550.in‚0^Õ 4¼6/ _wandbruntime‚23707  _runtime‚23707½³n¸Ì½6Æ trainer/global_step‚13049 % _timestamp‚1.7848629442941308e+09 _runtime‚23707.651404342 _step‚1086 ' lr-AdamW/pg1‚1.3830575632237934e-06 ' lr-AdamW/pg2‚1.3830575632237934e-06ú™5Ѿ6Ë ' lr-AdamW/pg1‚1.3830575632237934e-06 ' lr-AdamW/pg2‚1.3830575632237934e-06 trainer/global_step‚13049 % _timestamp‚1.7848629442941308e+09 _runtime‚23707.651404342 _step‚1086¾ªÇ4Œ4¿6/ _wandbruntime‚23707  _runtime‚23707`EÜ5(À6˜P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 _step‚1087 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113307630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1027997616493568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234494072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1044649035740160 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3769759532021955 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3757645428275200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234484002 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121186436 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3771263794355712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3771265165965312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121186436 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39778291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3771263794355712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3769760902061209 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1044649450904064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121176372 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3757645454448640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113307636 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚23707.655940754 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121176372 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13618366080512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74552031 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234494072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1044649035740160 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3769759532021955 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3757645428275200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234484002 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121186436 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3771263794355712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚355 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3771265165965312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121186436 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39778291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 415163904 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3771263794355712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3769760902061209 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚360 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1044649450904064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121176372 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3757645454448640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113307636 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚23707.655940754 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121176372 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13618366080512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74552031 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848629457666445e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3757924363739136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234511535 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113315997 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74557262 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3770039354597371 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13620734031872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121195532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113315997 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16653082025472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚13049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113316003 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1044728863436800 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39780887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16653028668416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3756442770226400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13596584370971 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13620734031872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3771545097771008 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234501451 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234511535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74557617 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34776380 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113316003 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3756442795654368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 _step‚1088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13597928982369 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13619388588544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121195532 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3757924337565696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34776375 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1028075834768384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39781237 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1044729284884992S?Ò-'Ä6§N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13620734031872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚350 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3771545097771008 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234501451 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234511535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74557617 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34776380 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113316003 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3756442795654368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 _step‚1088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13597928982369 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13619388588544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121195532 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3757924337565696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34776375 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1028075834768384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39781237 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1044729284884992 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7848629457666445e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3757924363739136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234511535 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113315997 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74557262 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3770039354597371 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13620734031872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121195532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 421448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113315997 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16653082025472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚13049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113316003 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1044728863436800 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39780887 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16653028668416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3756442770226400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13596584370971 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚355 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3771590918973440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3770104082569392 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3771590918973440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13669555636736 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1031926566700032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121642388 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74831445 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3785260448436736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39925710 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3783752094632817 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113725986 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 _step‚1091 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13669555636736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39926063 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1048640865521152 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235368374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16714298821120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34905735 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113725980 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121642388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚358 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13670901072896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1031926934791168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3771590892800000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74831803 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113725980 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121642388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚358 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13670901072896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1031926934791168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3771590892800000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74831803 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3771590918973440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3770104082569392 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3771590918973440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13669555636736 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1031926566700032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121642388 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74831445 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3785260448436736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39925710 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3783752094632817 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113725986 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 _step‚1091 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13669555636736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39926063 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1048640865521152 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235368374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16714298821120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34905735Ã5kHÑ4Ú6/ _wandbruntime‚23795  _runtime‚23795”2¼çw"Û6¡N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13671923588096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3770382884343168 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3771869828263936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13670578144768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3785541751852032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39928670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13671923588096 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1048721122533888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16715550116864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚23795.558303138 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1032005521157120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16715601376768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1032005153065984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121651484 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121641406 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74837045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34908380 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34908375 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113734347 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13670578144768 _step‚1092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3785540380235264 $ _timestamp‚1.784863032200578e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121651484 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235385837 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113734353 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113734347 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 HBDeviceStatsMonitúëô°or.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235385837 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1048720703182848 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39929016 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3784031917208345 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56å‘>\,'Ü6¦N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113734347 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13670578144768 _step‚1092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3785540380235264 $ _timestamp‚1.784863032200578e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121651484 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235385837 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113734353 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113734347 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235385837 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1048720703182848 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39929016 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3784031917208345 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13671923588096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3770382884343168 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3771869828263936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13670578144768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3785541751852032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39928670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13671923588096 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1048721122533888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16715550116864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚23795.558303138 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1032005521157120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16715601376768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1032005153065984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121651484 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121641406 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74837045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34908380 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34908375 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904Ä»cÊ4Ý6/ _wandbruntime‚23795  _runtime‚23795ººÁ Þ6‡ 3loss/FBt1/assignment_loss‚0.0015272399667024064 1loss/FBt2/assignment_loss‚0.01249841751852441 1loss/SRqqt1/detection_loss‚0.8800851525400466 epoch‚46 _runtime‚23795.595491434 0loss/FRt1/assignment_loss‚1.5493607781718841 2loss/SRqqt1/assignment_loss‚0.3863792051268093 /loss/FBt1/detection_loss‚0.5312032124867463 trainer/global_step‚13099 % _timestamp‚1.7848630322382784e+09 0loss/FRt2/assignment_loss‚0.7277386898503584 1loss/SRqqt2/detection_loss‚0.5953766415287346 /loss/FRt1/detection_loss‚0.5559026654124932 /loss/FRt2/detection_loss‚1.3580568799330313 /loss/FBt2/detection_loss‚0.3882296302565799 %loss/total_loss‚7.433853706690419 _step‚1093 2loss/SRqqt2/assignment_loss‚0.4474951938985084zrÉ ß6Œ 0loss/FRt2/assignment_loss‚0.7277386898503584 1loss/SRqqt2/detection_loss‚0.5953766415287346 /loss/FRt1/detection_loss‚0.5559026654124932 /loss/FRt2/detection_loss‚1.3580568799330313 /loss/FBt2/detection_loss‚0.3882296302565799 %loss/total_loss‚7.433853706690419 _step‚1093 2loss/SRqqt2/assignment_loss‚0.4474951938985084 3loss/FBt1/assignment_loss‚0.0015272399667024064 1loss/FBt2/assignment_loss‚0.01249841751852441 1loss/SRqqt1/detection_loss‚0.8800851525400466 epoch‚46 _runtime‚23795.595491434 0loss/FRt1/assignment_loss‚1.5493607781718841 2loss/SRqqt1/assignment_loss‚0.3863792051268093 /loss/FBt1/detection_loss‚0.5312032124867463 trainer/global_step‚13099 % _timestamp‚1.7848630322382784e+09Åâûžà6‚(:“ Á²‹ÓœÎÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.424' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.14133333333332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w²‹¯Iá6‚(:¾  Á²‹Ó¬›ú³ network.recv‚ 1.685439e+06 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0$ memory_percent‚92.14700063069662 proc.memory.rssMB‚ 2902.1875 disk.dm-0.in‚ 123.63671875 disk.drbd2111.out‚0" disk.dm-0.out‚57033.0380859375 disk.drbd1356.out‚0 disk.rbd1.out‚118.4716796875 disk.drbd1552.out‚0 disk.drbd1584.out‚0' proc.memory.availableMB‚ 1929.953125 disk.sda2.out‚ 925.015625 disk.drbd2111.in‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0% disk.nvme0n1.out‚57033.0380859375 network.sent‚ 2.292435e+07 disk.rbd1.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1673.in‚0) disk./.usagePercent‚70.51407140557482 disk.drbd1931.out‚0 disk.drbd2078.in‚0% disk./.usageGB‚1260.8632125854492 disk.drbd1965.in‚0 cpu‚25.340397374446155 disk.drbd2041.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1289.out‚0 disk.drbd1584.in‚0* proc.memory.percent‚11.809031168619793 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.nvme0n1.in‚ 123.63671875 disk.sda.out‚ 925.015625/ô’â6‚(:‡ в‹Ó§•È³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚181.593' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚60.531 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ôHCHã6‚(:½  в‹Ó±æç³ disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚925.3125 disk.drbd1965.in‚0 disk.drbd1285.in‚0% disk.nvme0n1.out‚57033.1982421875 cpu‚25.340340858352338* proc.memory.percent‚11.809031168619793 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.sda.out‚925.3125 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.drbd1552.in‚0 disk.sda.in‚ 0.0390625 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.8632125854492 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚118.4716796875 disk.drbd1226.out‚0 proc.memory.rssMB‚ 2902.1875 disk.drbd1550.out‚0 disk.drbd1356.out‚0" disk.dm-0.out‚57033.1982421875 disk.drbd2041.out‚0 disk.drbd2111.in‚0 network.recv‚ 1.688442e+06 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.sda2.in‚ 0.0390625 disk.drbd1828.out‚0 disk.dm-0.in‚ 123.63671875) proc.memory.availableMB‚ 1929.63671875 network.sent‚ 2.2990021e+07 disk.sda1.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0) disk./.usagePercent‚70.51407140557482$ memory_percent‚92.14828809102377cEÌ°ä6‚(:’ ß²‹Ó®ëʳ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.584' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.19466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0× …Må6‚(:  ß²‹ÓÔ¨è³ disk.drbd2078.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.14916229248047 disk.drbd1584.out‚0 disk.sda.in‚0.046875 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd2111.in‚0& disk.nvme0n1.out‚57033.83837890625 disk.drbd1226.in‚0 proc.cpu.threads‚17* proc.memory.percent‚11.809031168619793 network.recv‚ 1.688508e+06( disk./.usagePercent‚70.5140989261563 disk.drbd1817.out‚0 disk.sda2.out‚ 925.62890625 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0 proc.memory.rssMB‚ 2902.1875 disk.sda.out‚ 925.62890625 disk.drbd1965.in‚0 disk.drbd2041.in‚0 disk.drbd1510.out‚0' proc.memory.availableMB‚ 1929.421875 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1550.in‚0 cpu‚25.340305013130617 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1550.out‚0 disk.drbd1828.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd1938.out‚0 disk.drbd1285.out‚0% disk./.usageGB‚1260.8637046813965 disk.sda2.in‚0.046875 disk.dm-0.in‚ 123.63671875 disk.drbd1584.in‚0# disk.dm-0.out‚57033.83837890625 network.sent‚ 2.2990087e+07 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.rbd1.out‚118.6357421875 disk.drbd1510.in‚0Ùì”æ6‚(:’ ÓÔ›Ù³ gpu.0.gpu‚36.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.968' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.32266666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î)³øLç6‚(:Á  Óÿ­ú³ disk.nvme0n1.in‚ 123.63671875) disk./.usagePercent‚70.51409679277789 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.rbd1.out‚118.6357421875$ memory_percent‚92.14916229248047 cpu‚25.340258466285533 disk.drbd1289.out‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1931.in‚0 proc.cpu.threads‚17 network.sent‚ 2.2991168e+07 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.sda.out‚ 925.8828125 disk.sda2.in‚0.046875 disk.drbd1289.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0' proc.memory.availableMB‚ 1929.421875 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 925.8828125 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.8636665344238 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0# disk.dm-0.out‚57033.99853515625 disk.drbd1510.in‚0 disk.drbd1550.in‚0 proc.memory.rssMB‚ 2902.1875 disk.drbd1584.in‚0 disk.sda.in‚0.046875 disk.drbd1285.out‚0* proc.memory.percent‚11.809031168619793 network.recv‚ 1.689497e+06 disk.drbd2118.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1552.out‚0& disk.nvme0n1.out‚57033.99853515625 disk.drbd2041.out‚0 disk.drbd2111.in‚0 disk.drbd1285.in‚0AÉõNMè6‚(:  ý²‹ÓîçÙ³& disk.nvme0n1.out‚57034.29638671875 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0 network.sent‚ 2.2991234e+07 disk.drbd2041.in‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0( proc.memory.availableMB‚ 1929.6640625 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.046875 disk.drbd1289.in‚0 disk.drbd1938.in‚0 cpu‚25.340223292713496 disk.drbd2111.in‚0 disk.rbd1.out‚ 118.66796875 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0# disk.dm-0.out‚57034.29638671875 proc.memory.rssMB‚ 2902.1875 disk.drbd1552.out‚0 disk.sda2.in‚0.046875 disk.nvme0n1.in‚ 123.63671875 disk.drbd1510.in‚0 disk.drbd1673.out‚0 network.recv‚ 1.689563e+06 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 926.41015625* proc.memory.percent‚11.809031168619793 proc.cpu.threads‚17 disk.drbd1550.out‚0 disk.sda2.out‚ 926.41015625 disk.dm-0.in‚ 123.63671875% disk./.usageGB‚1260.8636665344238 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0$ memory_percent‚92.14817682902017 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0) disk./.usagePercent‚70.51409679277789 disk.drbd1931.out‚0ímµžé6‚(:“ ý²‹Ó̹³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.465' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.48833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸¦Xê6‚(:’ Œ³‹ÓË©³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚264.917' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.30566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å7õ¿Ië6‚(:¾  Œ³‹Óþú˳% disk./.usageGB‚1260.8636665344238 disk.sda.out‚ 926.5859375 disk.drbd2078.out‚0 disk.drbd2041.out‚0 proc.memory.rssMB‚ 2902.1875 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1510.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1584.out‚0$ memory_percent‚92.14919408162436 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.rbd1.out‚ 118.66796875 network.sent‚ 2.29913e+07 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd2041.in‚0 disk.drbd1226.out‚0( proc.memory.availableMB‚ 1929.4140625 disk.sda2.out‚ 926.5859375 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.046875 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1828.out‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.51409679277789 disk.drbd2078.in‚0& disk.nvme0n1.out‚57034.45654296875 disk.drbd2118.in‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.809031168619793 proc.cpu.threads‚17 network.recv‚ 1.689629e+06 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1965.out‚0 disk.drbd1552.out‚0 disk.sda1.out‚0 cpu‚25.340229660024846 disk.rbd1.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.046875# disk.dm-0.out‚57034.45654296875 disk.drbd2118.out‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0ðg°4ì6/ _wandbruntime‚23880  _runtime‚23880‹ÚûÌí6Æ _step‚1094 ' lr-AdamW/pg1‚1.1083591868899782e-06 ' lr-AdamW/pg2‚1.1083591868899782e-06 trainer/global_step‚13149 % _timestamp‚1.7848631175824318e+09 _runtime‚23880.939852689öxÔÑî6Ë trainer/global_step‚13149 % _timestamp‚1.7848631175824318e+09 _runtime‚23880.939852689 _step‚1094 ' lr-AdamW/pg1‚1.1083591868899782e-06 ' lr-AdamW/pg2‚1.1083591868899782e-06ÆûâU54ï6/ _wandbruntime‚23880  _runtime‚23880Á0¶‘Íð6˜P % _timestamp‚1.7848631175836456e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3799258474127360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122098340 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40073814 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1035856253088768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122098340 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3799257102517760 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3785536383498240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75111549 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3797743287204425 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3799258474127360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3784044145830224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13722090629120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236242676 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3799257102517760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114144330 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114144336 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13697771334947 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40074155 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3797741917165171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114144336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35037740 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3785536357324800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3785536383498240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13699115946233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236232606 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16776796703744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3785536357324800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122088276 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3784044171258192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13722090629120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1052632681701376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122088276 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1052633098962432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚13149 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35037735 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75111895 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚3419=‰Ã#(ñ6P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1095 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3785536357324800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122088276 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3784044171258192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13722090629120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49169920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1052632681701376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122088276 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1052633098962432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚13149 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35037735 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75111895 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚341 % _timestamp‚1.7848631175836456e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3799258474127360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122098340 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40073814 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1035856253088768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122098340 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3799257102517760 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3785536383498240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75111549 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3797743287204425 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3799258474127360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3784044145830224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13722090629120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236242676 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 417261056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3799257102517760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114144330 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114144336 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13697771334947 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40074155 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3797741917165171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114144336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35037740 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3785536357324800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3785536383498240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13699115946233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236232606 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16776796703744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3785815292788736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3785815266615296 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3785815292788736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3798021739740587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122097358 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35040380 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122107436 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114152697 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3799538405933056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236260139 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3784322947604000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236250055 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114152697 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35040375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40077066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122107436 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1035934839454720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1096 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3799538405933056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75117098 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16778096303616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3785815266615296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236260139 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236250055 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40076723 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1052712514310144F\®Î $ô6§N YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1052712514310144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 _runtime‚23882.412440079 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13723113144320 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3798023109779953 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13698792136587 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16778042946560 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13721767700992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3785815292788736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3785815266615296 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3785815292788736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3798021739740587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122097358 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35040380 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122107436 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114152697 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3799538405933056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236260139 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3784322947604000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236250055 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114152697 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35040375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40077066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122107436 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1035934839454720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53357056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1096 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3799538405933056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75117098 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16778096303616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3785815266615296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236260139 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236250055 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40076723Èëá%‚4õ6/ _wandbruntime‚23882  _runtime‚23882‘>¾‹ ö6‡ 1loss/SRqqt2/detection_loss‚0.5863871735635475 epoch‚46 /loss/FBt1/detection_loss‚0.6967970186061311 .loss/FBt2/detection_loss‚0.372886894967804 2loss/FBt2/assignment_loss‚0.037079016758239276 1loss/SRqqt1/detection_loss‚0.7963012651631531 _step‚1097 0loss/FRt1/assignment_loss‚1.5767594336524724 2loss/SRqqt1/assignment_loss‚0.4346837033408216 /loss/FRt1/detection_loss‚0.5511953862563761 %loss/total_loss‚7.621387532646166 trainer/global_step‚13149 % _timestamp‚1.7848631190943363e+09 _runtime‚23882.451514363 /loss/FRt2/assignment_loss‚0.629216458813279 3loss/SRqqt2/assignment_loss‚0.46480883800252687 4loss/FBt1/assignment_loss‚0.00033010546410736845 .loss/FRt2/detection_loss‚1.474942238057708!A[³÷6Œ .loss/FBt2/detection_loss‚0.372886894967804 2loss/FBt2/assignment_loss‚0.037079016758239276 1loss/SRqqt1/detection_loss‚0.7963012651631531 _step‚1097 0loss/FRt1/assignment_loss‚1.5767594336524724 2loss/SRqqt1/assignment_loss‚0.4346837033408216 /loss/FRt1/detection_loss‚0.5511953862563761 %loss/total_loss‚7.621387532646166 trainer/global_step‚13149 % _timestamp‚1.7848631190943363e+09 _runtime‚23882.451514363 /loss/FRt2/assignment_loss‚0.629216458813279 3loss/SRqqt2/assignment_loss‚0.46480883800252687 4loss/FBt1/assignment_loss‚0.00033010546410736845 .loss/FRt2/detection_loss‚1.474942238057708 1loss/SRqqt2/detection_loss‚0.5863871735635475 epoch‚46 /loss/FBt1/detection_loss‚0.6967970186061311Éa9 ›žø6‚(:“ ›³‹ÓÅû¿³ gpu.0.gpu‚97.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.511' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.17033333333333 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wÜ(üMù6‚(:  ›³‹Ó¢«â³ disk.drbd2078.out‚0 disk.drbd2111.in‚0 proc.memory.rssMB‚ 2902.1875 disk.drbd1584.in‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.809031168619793 disk.nvme0n1.in‚ 123.63671875 disk.sda.out‚ 926.99609375) disk./.usagePercent‚70.51409679277789 disk.drbd1356.out‚0 disk.drbd1828.in‚0 disk.drbd1931.in‚0# disk.dm-0.out‚57034.97802734375 disk.drbd2041.in‚0 disk.sda.in‚0.046875 disk.drbd1285.out‚0& disk.nvme0n1.out‚57034.97802734375 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.rbd1.out‚ 118.83203125 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0( proc.memory.availableMB‚ 1929.3515625 cpu‚25.340183156883704 disk.dm-0.in‚ 123.63671875 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0$ memory_percent‚92.14944839477539 disk.drbd1226.out‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0.046875 disk.sda2.out‚ 926.99609375 proc.cpu.threads‚17 disk.drbd1289.out‚0 disk.drbd1817.in‚0% disk./.usageGB‚1260.8636665344238 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 network.recv‚ 1.692701e+06 disk.drbd1356.in‚0 network.sent‚ 2.3061657e+07 disk.drbd1550.in‚0 disk.drbd1938.in‚0Z o›ú6‚(: ª³‹Ó£óÛ³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚339.13' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚113.04333333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼(RåOû6‚(:Ä  ª³‹Ó­ãù³ disk.nvme0n1.in‚ 123.63671875 disk.drbd1817.out‚0 disk.drbd1510.in‚0 disk.drbd1931.out‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0$ memory_percent‚92.19420750935873 disk.drbd1510.out‚0( proc.memory.availableMB‚ 1918.3515625 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 927.3359375 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0# proc.memory.rssMB‚ 2912.99609375 cpu‚25.341170379688833 network.recv‚ 1.692809e+06& disk.nvme0n1.out‚57035.13037109375 disk.drbd2118.in‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.8636665344238 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.853011449178059 network.sent‚ 2.3061765e+07 disk.drbd1552.in‚0 disk.sda2.in‚0.046875) disk./.usagePercent‚70.51409679277789 disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.sda.out‚ 927.3359375 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 118.83203125 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.sda.in‚0.046875# disk.dm-0.out‚57035.13037109375 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0Òºƒ4ü6/ _wandbruntime‚23919  _runtime‚23919ÍR€æ;ý6ûu 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35604768991470337 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5906494855880737 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7120441794395447 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2986522614955902 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.052645765244960785 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6535841822624207 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.35688069462776184 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35604768991470337 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.034030791372060776 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38541001081466675 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02855331264436245 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2801506519317627 particle/accuracy_1_of_4‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.666343629360199 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6142292022705078 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0863361731171608 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.883858859539032 'particle/f_score‚0.6549786329269409 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27984997630119324 particle/accuracy_2_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16068461537361145 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36974623799324036 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17166413366794586 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.707310140132904 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7798714637756348 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7967861890792847 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 *jet/accuracy_2_of_4‚0.8472596406936646 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4611905515193939 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 trainer/global_step‚13159 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22651158273220062 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07001408189535141 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7975193858146667 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7195653319358826 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7587496638298035 particle/accuracy_1_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 *jet/accuracy_1_of_3‚0.9374781250953674 6validation_average_jet_accuracy‚0.4060181677341461 epoch‚46 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6771191954612732 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5121767520904541 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6934900283813477 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8081672787666321 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26818209886550903 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29531246423721313 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34715184569358826 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9231768250465393 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18946368992328644 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6634291410446167 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02910783514380455 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2710246443748474 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6314156651496887 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7206862568855286 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7112331390380859 /particle/accuracy_4_of_4‚0.9676654934883118 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6388741135597229 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3674623668193817 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03482526168227196 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6578465104103088 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31020426750183105 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 +particle/specificity‚0.9126912951469421 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3851308226585388 /particle/accuracy_3_of_3‚0.9993678331375122 % _timestamp‚1.7848631558452876e+09 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3682210445404053 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7170284390449524 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22300377488136292 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5510358214378357 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 +particle/sensitivity‚0.6426438093185425 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _runtime‚23919.203362284 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38253840804100037 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6925764083862305 *jet/accuracy_2_of_2‚0.1722869873046875 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35625559091567993 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 )loss/val_total_loss‚7.713598926157617 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6910535097122192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.173086479306221 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6440390944480896 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7001362442970276 _step‚1098 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6880900263786316 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 (particle/accuracy‚0.8547537922859192 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35448554158210754 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7749959826469421 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8030920624732971 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14072434604167938 *jet/accuracy_1_of_2‚0.7900816202163696 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6481747627258301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8764428496360779 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6440390944480896 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40540432929992676 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6649436950683594 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04334162920713425 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07885482162237167 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3872714340686798 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9832039475440979 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7254734635353088 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_2_of_3‚0.6075951457023621 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17734672129154205 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6322289705276489 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7705167531967163 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6040360331535339 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6404638886451721 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5223209261894226 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2880726754665375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 +jet/accuracy_3_of_4‚0.10971427708864212 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30773335695266724 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_3‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26983457803726196 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.753845751285553 +jet/accuracy_1_of_1‚0.37198352813720703 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6345072984695435 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505a6+þ6€v 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27984997630119324 particle/accuracy_2_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16068461537361145 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.36974623799324036 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17166413366794586 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.707310140132904 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7798714637756348 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7967861890792847 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 *jet/accuracy_2_of_4‚0.8472596406936646 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4611905515193939 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 trainer/global_step‚13159 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22651158273220062 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07001408189535141 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7975193858146667 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7195653319358826 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7587496638298035 particle/accuracy_1_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 *jet/accuracy_1_of_3‚0.9374781250953674 6validation_average_jet_accuracy‚0.4060181677341461 epoch‚46 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6771191954612732 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5121767520904541 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6934900283813477 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8081672787666321 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26818209886550903 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29531246423721313 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34715184569358826 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9231768250465393 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18946368992328644 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6634291410446167 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02910783514380455 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2710246443748474 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6314156651496887 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7206862568855286 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7112331390380859 /particle/accuracy_4_of_4‚0.9676654934883118 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6388741135597229 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3674623668193817 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03482526168227196 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6578465104103088 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31020426750183105 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 +particle/specificity‚0.9126912951469421 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3851308226585388 /particle/accuracy_3_of_3‚0.9993678331375122 % _timestamp‚1.7848631558452876e+09 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3682210445404053 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7170284390449524 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22300377488136292 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5510358214378357 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.883858859539032 +particle/sensitivity‚0.6426438093185425 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 _runtime‚23919.203362284 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38253840804100037 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6925764083862305 *jet/accuracy_2_of_2‚0.1722869873046875 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35625559091567993 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 )loss/val_total_loss‚7.713598926157617 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6910535097122192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.173086479306221 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6440390944480896 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7001362442970276 _step‚1098 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6880900263786316 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 (particle/accuracy‚0.8547537922859192 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35448554158210754 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7749959826469421 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8030920624732971 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14072434604167938 *jet/accuracy_1_of_2‚0.7900816202163696 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6481747627258301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8764428496360779 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6440390944480896 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40540432929992676 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6649436950683594 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04334162920713425 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4791666865348816 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07885482162237167 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3872714340686798 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9832039475440979 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7254734635353088 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 =&Purity/0FRt0åöeqÐSRqqt0FBt/event_proportion‚0.0469098761677742 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 *jet/accuracy_2_of_3‚0.6075951457023621 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17734672129154205 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6322289705276489 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 * Purity/1FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7705167531967163 * Purity/*FRt2SRqqt1FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6040360331535339 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6404638886451721 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5223209261894226 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2880726754665375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 +jet/accuracy_3_of_4‚0.10971427708864212 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30773335695266724 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_3‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26983457803726196 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.753845751285553 +jet/accuracy_1_of_1‚0.37198352813720703 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6345072984695435 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35604768991470337 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5906494855880737 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7120441794395447 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2986522614955902 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.052645765244960785 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6535841822624207 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.35688069462776184 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35604768991470337 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4679487347602844 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.034030791372060776 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38541001081466675 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02855331264436245 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2801506519317627 particle/accuracy_1_of_4‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.666343629360199 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6142292022705078 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0863361731171608 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.883858859539032 'particle/f_score‚0.6549786329269409ʼ# “ÿ6‚(:ˆ ¹³‹Óˆ±Ç³ gpu.0.gpu‚62.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.943' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.981 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€!{óI€7‚(:¾  ¹³‹Ó—ê³ disk.drbd2111.in‚0 disk.drbd1510.out‚0 disk.sda2.in‚0.046875) proc.memory.availableMB‚ 1917.20703125 proc.memory.rssMB‚ 2914.3125" disk.dm-0.out‚57035.4462890625 disk.drbd2041.out‚0% disk.nvme0n1.out‚57035.4462890625 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1289.in‚0$ memory_percent‚92.19886461893717 disk.nvme0n1.in‚ 123.63671875 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd1817.in‚0$ disk./.usageGB‚1260.863754272461 disk.drbd1356.out‚0 disk.sda.out‚ 927.8515625 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.sda2.out‚ 927.8515625) disk./.usagePercent‚70.51410169954822 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd2118.in‚0 disk.drbd1673.out‚0 network.recv‚ 1.695625e+06 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 119.07421875 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1226.out‚0 network.sent‚ 2.3118759e+07 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.046875 cpu‚25.342146595852604 disk.drbd1289.out‚0* proc.memory.percent‚11.858367919921875 proc.cpu.threads‚17 disk.drbd2111.out‚0 disk.drbd2078.out‚0 disk.drbd1550.out‚0Ò].œ7‚(:‘ ȳ‹ÓãÄܳ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚275.77' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.92333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êVYëG‚7‚(:¼  ȳ‹Óœü³ disk.drbd1828.in‚0 disk.rbd1.out‚ 119.07421875 disk.drbd1584.in‚0 disk.sda2.in‚0.046875 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0" disk.dm-0.out‚57035.8623046875) disk./.usagePercent‚70.51410169954822 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1550.in‚0 cpu‚25.342088630186144 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 proc.memory.rssMB‚ 2914.3125* proc.memory.percent‚11.858367919921875 network.recv‚ 1.695733e+06 disk.drbd2078.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0% disk.nvme0n1.out‚57035.8623046875 disk.sda2.out‚ 928.171875$ disk./.usageGB‚1260.863754272461 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 123.63671875 disk.sda1.in‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1226.in‚0 disk.sda.out‚ 928.171875$ memory_percent‚92.20032691955566 disk.drbd1938.in‚0) proc.memory.availableMB‚ 1916.84765625 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.046875 network.sent‚ 2.3118867e+07 disk.drbd1550.out‚0 disk.drbd1673.in‚0 ªÒ0ƒ7‚(:’ ׳‹ÓÁèâ³ gpu.0.gpu‚87.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚263.366' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.78866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$üBFG„7‚(:¼  ׳‹Óèóü³ proc.cpu.threads‚17$ disk./.usageGB‚1260.863754272461 disk.drbd1356.in‚0% disk.nvme0n1.out‚57036.5322265625 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0& proc.memory.availableMB‚ 1916.84375 network.recv‚ 1.695799e+06 disk.drbd2041.in‚0 disk.sda.out‚ 928.48046875* proc.memory.percent‚11.858367919921875 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 123.63671875) disk./.usagePercent‚70.51410169954822 disk.sda1.in‚0 disk.sda2.out‚ 928.48046875 proc.memory.rssMB‚ 2914.3125 cpu‚25.342072719101626 network.sent‚ 2.3118933e+07 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.sda2.in‚0.046875 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1817.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1938.out‚0# memory_percent‚92.2003428141276 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0" disk.dm-0.out‚57036.5322265625 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd2111.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 119.17578125 disk.drbd2118.in‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.046875ZÅ*‰C…7‚(:¸  泋Ó“¨ã³ disk.sda2.out‚ 928.8203125 disk.rbd1.out‚ 119.23828125 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0* proc.memory.percent‚11.858367919921875 cpu‚25.34201562477786 disk.drbd2111.out‚0 disk.sda.in‚0.046875) disk./.usagePercent‚70.51410276623743 disk.drbd2041.out‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 928.8203125 disk.drbd1931.out‚0 disk.drbd1356.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 network.recv‚ 1.696831e+06 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1550.out‚0! disk.dm-0.out‚57037.275390625 network.sent‚ 2.3120124e+07% disk./.usageGB‚1260.8637733459473 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2914.3125 disk.sda2.in‚0.046875 disk.dm-0.in‚ 123.63671875 disk.drbd2078.in‚0 disk.drbd1817.in‚0 disk.drbd1965.out‚0$ disk.nvme0n1.out‚57037.275390625# memory_percent‚92.2003428141276& proc.memory.availableMB‚ 1916.84375 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0†7‚(:’ 泋ÓýÕų gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚220.132' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.37733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)+[Ї7‚(:„ õ³‹Ó•ü׳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.9' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.3 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÈ©DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16846669100032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122591535 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3802042498574336 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1056681239999488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13755828668350 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114652118 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122591535 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3814297719727256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784863224230583e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1056680818544128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114652112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13778898621952 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114652118 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 _step‚1100 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3802042524747776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16846615735808 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13777553185792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13777553185792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3800543235670192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 trainer/global_step‚13199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3815820051760128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3802042524747776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 _runtime‚23987.588078201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114652112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40249271 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3815821423369728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35176632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3800543261098160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3815821423369728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16846669100032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122591535 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3802042498574336 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1056681239999488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13755828668350 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114652118 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122591535 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3814297719727256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784863224230583e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1056680818544128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114652112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13778898621952 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114652118 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚342 _step‚1100 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3802042524747776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16846615735808 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13777553185792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13777553185792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 421455360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3800543235670192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 trainer/global_step‚13199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3815820051760128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3802042524747776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 _runtime‚23987.588078201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114652112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40249271 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3815821423369728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 XKDeviceStatsMonitor.on_train_batch_staN2 FÚ rt/inactive_split.large_pool.allocated‚35176632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3800543261098160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3815821423369728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3802321434038272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13778575693824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3816099983558656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40252209 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1101 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 % _timestamp‚1.7848632257076802e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3802321434038272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13779921137152 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114660479 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1056761073981440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3814577542302672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114660485 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114660485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13779921137152 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122600617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13756849470102 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3816101355175424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75431476 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3814578912342038 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237261096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75431825 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1039912789174272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16847916716032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13755504858704 _runtime‚23989.065261885 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271180 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1056760642047488 trainer/global_step‚13199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13778575693824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35179267 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122610695 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 % _timestamp‚1.7848632257076802e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3802321434038272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13779921137152 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚63842816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114660479 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1056761073981440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 431933952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3814577542302672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114660485 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114660485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13779921137152 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122600617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13756849470102 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3816101355175424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75431476 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3814578912342038 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237261096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75431825 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1039912789174272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚349 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16847916716032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13755504858704 _runtime‚23989.065261885 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271180 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1056760642047488 trainer/global_step‚13199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13778575693824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35179267 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122610695 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299ÍÄþˆ24’7/ _wandbruntime‚23989  _runtime‚23989Åg- “7† 1loss/FBt2/assignment_loss‚0.07770036600729457 /loss/FRt2/detection_loss‚1.4120820964306864 %loss/total_loss‚7.700037998098582 trainer/global_step‚13199 1loss/SRqqt1/assignment_loss‚0.509465072000943 3loss/SRqqt2/assignment_loss‚0.46995294593654585 epoch‚47 _runtime‚23989.102597772 0loss/FRt1/assignment_loss‚1.5927401625669444 0loss/FRt2/assignment_loss‚0.6032648495981626 3loss/FBt1/assignment_loss‚5.602029446226776e-05 1loss/SRqqt2/detection_loss‚0.6372303047953823 .loss/FBt2/detection_loss‚0.409450598942642 _step‚1102 /loss/FRt1/detection_loss‚0.5118095590245663 1loss/SRqqt1/detection_loss‚0.7732095033638536 /loss/FBt1/detection_loss‚0.7030765191370985 % _timestamp‚1.7848632257453806e+09Žc%”7‹ 1loss/FBt2/assignment_loss‚0.07770036600729457 /loss/FRt2/detection_loss‚1.4120820964306864 %loss/total_loss‚7.700037998098582 trainer/global_step‚13199 1loss/SRqqt1/assignment_loss‚0.509465072000943 3loss/SRqqt2/assignment_loss‚0.46995294593654585 epoch‚47 _runtime‚23989.102597772 0loss/FRt1/assignment_loss‚1.5927401625669444 0loss/FRt2/assignment_loss‚0.6032648495981626 3loss/FBt1/assignment_loss‚5.602029446226776e-05 1loss/SRqqt2/detection_loss‚0.6372303047953823 .loss/FBt2/detection_loss‚0.409450598942642 _step‚1102 /loss/FRt1/detection_loss‚0.5118095590245663 1loss/SRqqt1/detection_loss‚0.7732095033638536 /loss/FBt1/detection_loss‚0.7030765191370985 % _timestamp‚1.7848632257453806e+09Î*á!Ç›•7‚(: „´‹Ó¼€×³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.567' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.85566666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 &ÉöO–7‚(:Ä  „´‹Ó”ó÷³& disk.nvme0n1.out‚57038.20068359375 proc.memory.rssMB‚ 2914.3125 disk.drbd1931.in‚0 disk.drbd1828.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0( proc.memory.availableMB‚ 1916.8046875 disk.drbd1552.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1550.in‚0 disk.sda.out‚ 929.47265625 disk.sda2.out‚ 929.47265625 disk.drbd1931.out‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.8637771606445 disk.dm-0.in‚ 123.63671875 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.046875 disk.drbd2041.in‚0$ memory_percent‚92.20050175984701 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚119.2705078125 cpu‚25.341942038544037 network.sent‚ 2.3120298e+07 disk.drbd1550.out‚0 disk.sda2.in‚0.046875 network.recv‚ 1.697005e+06# disk.dm-0.out‚57038.20068359375 disk.drbd1356.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.858367919921875 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.51410297957528 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0½æ=—7‚(:’ “´‹ÓèƳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚299.453' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.81766666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bÞìàM˜7‚(:  “´‹Óº¥ä³ disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.8647346496582 network.recv‚ 1.698048e+06 disk.drbd1828.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd2078.out‚0 network.sent‚ 2.3128007e+07% disk.nvme0n1.out‚57038.6982421875 disk.dm-0.in‚ 123.63671875* proc.memory.percent‚11.858367919921875 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.sda2.in‚0.046875 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1226.in‚0 disk.drbd1552.out‚0" disk.dm-0.out‚57038.6982421875 disk.drbd1356.out‚0 disk.rbd1.out‚119.5517578125 disk.drbd1226.out‚0 disk.drbd1965.in‚0 cpu‚25.341895407188023 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1938.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.rbd1.in‚0 disk.drbd2118.in‚0$ memory_percent‚92.20050175984701( proc.memory.availableMB‚ 1916.8046875 disk.drbd1550.out‚0 disk.drbd1938.in‚0 proc.memory.rssMB‚ 2914.3125 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 929.79296875 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.sda.in‚0.046875 disk.drbd1673.in‚0 disk.drbd1931.in‚0) disk./.usagePercent‚70.51415652737333 disk.sda2.out‚ 929.79296875 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0§^º™7‚(:’ ¢´‹ÓúØÔ³ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.803' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.26766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ´ˆLš7‚(:Á  ¢´‹Óúäô³ disk.drbd1673.out‚0 cpu‚25.34185861540931 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.046875 disk.drbd2078.out‚0* proc.memory.percent‚11.858367919921875 network.recv‚ 1.698198e+06 disk.drbd1510.out‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 proc.memory.rssMB‚ 2914.3125 disk.drbd1828.in‚0% disk.nvme0n1.out‚57038.8310546875 disk.drbd1584.out‚0 disk.sda.out‚ 930.05078125 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 123.63671875$ memory_percent‚92.20050175984701 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 930.05078125( proc.memory.availableMB‚ 1916.8046875) disk./.usagePercent‚70.51415652737333% disk./.usageGB‚1260.8647346496582 disk.drbd1285.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1965.out‚0 network.sent‚ 2.3128157e+07 disk.drbd2041.out‚0 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.drbd1550.out‚0" disk.dm-0.out‚57038.8310546875 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.sda2.in‚0.046875 disk.drbd1356.in‚0 disk.rbd1.out‚119.6142578125 disk.drbd1550.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0Ží:œ›7‚(:‘ ±´‹Ó•«¶³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚341.477' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚113.82566666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Í@œ9œ7‚(:®  ±´‹Ó˜ÉÖ³ disk.drbd2111.in‚0 disk.drbd1673.out‚0 disk.drbd2041.out‚0 disk.sda2.out‚930.5 disk.rbd1.in‚0 disk.rbd1.out‚119.6142578125 network.recv‚ 1.698264e+06 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.drbd1931.in‚0 cpu‚25.341812297514636% disk./.usageGB‚1260.8647346496582 disk.drbd1226.out‚0 disk.drbd1552.out‚0 disk.sda.out‚930.5$ memory_percent‚92.20050175984701 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 57039.2109375 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 network.sent‚ 2.3128223e+07) disk./.usagePercent‚70.51415652737333 disk.drbd1550.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.858367919921875 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 proc.memory.rssMB‚ 2914.3125 disk.sda2.in‚0.046875 disk.drbd1965.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.sda.in‚0.046875 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2078.in‚0" disk.nvme0n1.out‚ 57039.2109375 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd1828.out‚0( proc.memory.availableMB‚ 1916.8046875 disk.drbd1226.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.dm-0.in‚ 123.63671875KlÞÃJ7‚(:’ À´‹ÓĶ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚66.0 gpu.0.powerWatts‚288.134' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.04466666666667 gpu.0.smClock‚1410 gpu.0.memoryCï ‰,Slock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dšQGž7‚(:¼  À´‹Ó»äÕ³ disk.drbd2118.out‚0 disk.sda2.out‚ 930.80859375 network.sent‚ 2.3193678e+07 disk.drbd1584.in‚0 disk.sda.in‚0.046875% disk./.usageGB‚1260.8647346496582 disk.drbd1931.out‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1965.out‚0 network.recv‚ 1.700147e+06 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 disk.sda1.in‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0( proc.memory.availableMB‚ 1918.4296875 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.858367919921875 disk.sda2.in‚0.046875 disk.drbd1931.in‚0 disk.dm-0.out‚57039.37109375 disk.drbd2041.out‚0 cpu‚25.34175515468666) disk./.usagePercent‚70.51415652737333 disk.nvme0n1.in‚ 123.63671875 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.rbd1.out‚ 119.646484375 disk.drbd1510.in‚0 disk.drbd1828.in‚0$ memory_percent‚92.19388961791992 proc.cpu.threads‚17 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 proc.memory.rssMB‚ 2914.3125# disk.nvme0n1.out‚57039.37109375 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 930.80859375 disk.drbd2111.in‚0–'[44Ÿ7/ _wandbruntime‚24074  _runtime‚24074£ÉîÑÊ 7Ä & lr-AdamW/pg1‚8.637845641308166e-07 & lr-AdamW/pg2‚8.637845641308166e-07 trainer/global_step‚13249 % _timestamp‚1.7848633108488386e+09 _runtime‚24074.206090131 _step‚1103Å>ãÏ¡7É trainer/global_step‚13249 % _timestamp‚1.7848633108488386e+09 _runtime‚24074.206090131 _step‚1103 & lr-AdamW/pg1‚8.637845641308166e-07 & lr-AdamW/pg2‚8.637845641308166e-07Ï—"¹4¢7/ _wandbruntime‚24074  _runtime‚24074¢Usk(£7—P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3829818077450752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40396097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13805587939872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75704724 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13828742742016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3829818077450752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1043763889197056 _runtime‚24074.21091105 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3815987963099136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3828288912298864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123047487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13806932551158 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13830088178176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1060672879536128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13828742742016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238128019 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13830088178176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3829816705841152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16908939072000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35308627 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1060672460177920 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3828290282338118 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115070468 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75705085 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16908990339072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3815987989272576 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1043763521105920 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3815987989272576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123057551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3814483349786960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968h“ÙX"(¤7œP bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1043763521105920 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3815987989272576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123057551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3814483349786960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3829818077450752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40396097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13805587939872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75704724 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13828742742016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3829818077450752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1043763889197056 _runtime‚24074.21091105 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3815987963099136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3828288912298864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123047487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13806932551158 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13830088178176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1060672879536128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13828742742016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238128019 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13830088178176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3829816705841152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16908939072000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35308627 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1060672460177920 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3828290282338118 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115070468 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75705085 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16908990339072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3815987989272576ÐLŽë+’¥7‚(:‡ Ï´‹Ó§ž®³ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.665' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l¸DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1105 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238145482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060752719480832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75710640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123056569 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123066647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3830096637639680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3830098009256448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75710280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 `KDeviceStatsMonitor.on_train_batch_end/inactive_spï'eflit_bytes.large_pool.freed‚1043842107471872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115078835 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13807953352910 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123056569 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115078829®‹‚M-'©7§N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35311272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40399368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13831110693376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚355 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1105 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238145482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060752719480832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75710640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123056569 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123066647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3830096637639680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3830098009256448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75710280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1043842107471872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115078835 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13807953352910 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123056569 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115078829 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3814762151560736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3828568734874280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13829765250048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16910188463616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264ÑΧ½¶4ª7/ _wandbruntime‚24075  _runtime‚24075Ü6Œ«7Š 0loss/FRt1/assignment_loss‚1.5765418716009845 3loss/SRqqt1/assignment_loss‚0.44039029244660316 2loss/SRqqt2/assignment_loss‚0.4178641695274658 4loss/FBt1/assignment_loss‚0.00025088596845836435 trainer/global_step‚13249 _runtime‚24075.717569976 /loss/FRt2/detection_loss‚1.4654826930058507 /loss/FBt1/detection_loss‚0.4881754690372459 %loss/total_loss‚7.517695240144352 % _timestamp‚1.7848633123603077e+09 2loss/FBt2/assignment_loss‚0.017805589153470303 1loss/SRqqt2/detection_loss‚0.6362907318962497 /loss/FBt2/detection_loss‚0.4584476537347116 0loss/FRt2/assignment_loss‚0.5952925997254737 /loss/FRt1/detection_loss‚0.5235443422505154 1loss/SRqqt1/detection_loss‚0.8976089417973223 epoch‚47 _step‚1106ug.Û¬7 /loss/FBt2/detection_loss‚0.4584476537347116 0loss/FRt2/assignment_loss‚0.5952925997254737 /loss/FRt1/detection_loss‚0.5235443422505154 1loss/SRqqt1/detection_loss‚0.8976089417973223 epoch‚47 _step‚1106 0loss/FRt1/assignment_loss‚1.5765418716009845 3loss/SRqqt1/assignment_loss‚0.44039029244660316 2loss/SRqqt2/assignment_loss‚0.4178641695274658 4loss/FBt1/assignment_loss‚0.00025088596845836435 trainer/global_step‚13249 _runtime‚24075.717569976 /loss/FRt2/detection_loss‚1.4654826930058507 /loss/FBt1/detection_loss‚0.4881754690372459 %loss/total_loss‚7.517695240144352 % _timestamp‚1.7848633123603077e+09 2loss/FBt2/assignment_loss‚0.017805589153470303 1loss/SRqqt2/detection_loss‚0.6362907318962497ÒÔ”‘)ž­7‚(:“ Þ´‹ÓÞ¼¸³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚319.232' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚106.41066666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*4˜M®7‚(:  Þ´‹Ó‡‘׳% disk./.usageGB‚1260.8637771606445 disk.sda1.in‚0 disk.drbd1673.out‚0$ memory_percent‚92.19414393107095 cpu‚25.34170320333605) disk./.usagePercent‚70.51410297957528 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1552.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.046875 disk.dm-0.in‚ 123.63671875 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd1550.out‚0 network.sent‚ 2.3264814e+07 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 931.37109375 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.sda.in‚0.046875* proc.memory.percent‚11.858367919921875 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1965.in‚0 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 network.recv‚ 1.703457e+06 disk.rbd1.out‚ 119.896484375 disk.drbd1285.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚57039.90478515625 disk.sda2.out‚ 931.37109375 disk.drbd1356.out‚0 disk.drbd1817.in‚0 disk.drbd1938.out‚0# disk.dm-0.out‚57039.90478515625 proc.memory.rssMB‚ 2914.3125 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 1918.3671875 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd2078.out‚0-DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115488812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3829933427623936 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚1108 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239002321 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16971128138752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3829933427623936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13879932298240 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3843813359922176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75983598 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3828423438475760 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3829933453797376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3843814731531776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115488818 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123513503 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16971064288768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚391Ysnº#(¼7P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚391 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1064664335633408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚63849984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 _runtime‚24161.231938267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3842281474909726 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75983994 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115488812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3829933427623936 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 _step‚1108 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239002321 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16971128138752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3829933427623936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13879932298240 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3843813359922176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75983598 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3828423438475760 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3829933453797376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3843814731531776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115488818 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123513503 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16971064288768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Ô–°cÛ4½7/ _wandbruntime‚24162  _runtime‚24162ùy|''¾7¡N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35443267 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3828702240249536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115497185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115497179 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13857712624320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3842561297485254 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3844094663337472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123522599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3830212363087872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115497179 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239009700 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚382 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047771793860608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13882300249600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40546247 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3842559927445888 % _timestamp‚1.7848633993492353e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123512521 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75989137 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚24162.70686363 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16972362447872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13859057235718 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239019784 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13880954806272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚13299 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0üÕ{¸,'¿7¦N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13859057235718 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239019784 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13880954806272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚13299 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35443267 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3828702240249536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115497185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115497179 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13857712624320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3842561297485254 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3844094663337472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123522599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3830212363087872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115497179 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239009700 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚382 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047771793860608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13882300249600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40546247 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3842559927445888 % _timestamp‚1.7848633993492353e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123512521 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75989137 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _runtime‚24162.70686363 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16972362447872Õ÷Ê4À7/ _wandbruntime‚24162  _runtime‚24162íÏ:ÍÁ7ˆ 1loss/FBt1/assignment_loss‚0.04370719851635604 /loss/FRt2/detection_loss‚1.4244499308269907 epoch‚47 % _timestamp‚1.7848633993857634e+09 0loss/FRt2/assignment_loss‚0.656932251875àd89A4804 /loss/FRt1/detection_loss‚0.5143062303211919 0loss/FBt2/detection_loss‚0.42055234366441857 trainer/global_step‚13299 3loss/SRqqt2/assignment_loss‚0.48155135178174246 1loss/SRqqt1/detection_loss‚0.8896549041871531 1loss/SRqqt2/detection_loss‚0.5468008797174135 /loss/FBt1/detection_loss‚0.6252636753358982 %loss/total_loss‚7.673022168335557 _step‚1110 /loss/FRt1/assignment_loss‚1.608252486549282 3loss/SRqqt1/assignment_loss‚0.45670227552040993 2loss/FBt2/assignment_loss‚0.004848640039219608 _runtime‚24162.743063198±ër»Â7 /loss/FRt1/assignment_loss‚1.608252486549282 3loss/SRqqt1/assignment_loss‚0.45670227552040993 2loss/FBt2/assignment_loss‚0.004848640039219608 _runtime‚24162.743063198 1loss/FBt1/assignment_loss‚0.04370719851635604 /loss/FRt2/detection_loss‚1.4244499308269907 epoch‚47 % _timestamp‚1.7848633993857634e+09 0loss/FRt2/assignment_loss‚0.6569322518754804 /loss/FRt1/detection_loss‚0.5143062303211919 0loss/FBt2/detection_loss‚0.42055234366441857 trainer/global_step‚13299 3loss/SRqqt2/assignment_loss‚0.48155135178174246 1loss/SRqqt1/detection_loss‚0.8896549041871531 1loss/SRqqt2/detection_loss‚0.5468008797174135 /loss/FBt1/detection_loss‚0.6252636753358982 %loss/total_loss‚7.673022168335557 _step‚1110Ö&h„LÃ7‚(:Á  ©µ‹Ó†“ô³ disk.drbd1817.out‚0 disk.sda2.in‚0.046875 disk.drbd1510.in‚0) disk./.usagePercent‚70.51415226061651 disk.sda2.out‚ 933.1015625 disk.drbd2111.out‚0& disk.nvme0n1.out‚57042.39306640625 disk.sda1.out‚0 disk.drbd1584.in‚0 network.sent‚ 2.3277084e+07 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 network.recv‚ 1.707474e+06 disk.sda.out‚ 933.1015625 disk.drbd2078.in‚0 disk.sda.in‚0.046875 disk.drbd1931.out‚0 disk.rbd1.out‚120.0927734375$ disk./.usageGB‚1260.864658355713 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0$ memory_percent‚92.19840367635092( proc.memory.availableMB‚ 1917.3203125# disk.dm-0.out‚57042.39306640625 disk.drbd1285.in‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd2078.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.858367919921875 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 proc.memory.rssMB‚ 2914.3125 cpu‚25.341490807706855 disk.drbd1289.in‚0 disk.drbd1931.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1226.in‚0 disk.drbd1552.out‚0¸2Ò+œÄ7‚(:‘ ©µ‹Ó†Ô¦· gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.984' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.66133333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç! ‡“Å7‚(:ˆ ¸µ‹ÓÓÜܳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚301.995' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sb܉DÆ7‚(:¹  ¸µ‹Ó¦Ùû³ disk.drbd1817.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.sda2.out‚ 933.41796875 disk.drbd1931.in‚0 disk.drbd2041.in‚0 network.recv‚ 1.70754e+06 disk.drbd1817.in‚0 disk.drbd1285.in‚0 disk.drbd1584.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1550.out‚0# memory_percent‚92.1984831492106 network.sent‚ 2.327715e+07& disk.nvme0n1.out‚57042.65087890625 disk.drbd1584.in‚0 disk.rbd1.out‚120.125 disk.drbd1510.out‚0* proc.memory.percent‚11.858367919921875 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 cpu‚25.34144432262297 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 123.63671875# disk.dm-0.out‚57042.65087890625 disk.drbd2111.in‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.046875 disk.drbd2041.out‚0 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1356.in‚0 disk.sda.out‚ 933.41796875 disk.drbd1673.in‚0 disk.drbd1938.out‚0( disk./.usagePercent‚70.5141511939273 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2914.3125 disk.drbd2078.in‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.046875 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd1828.in‚0) proc.memory.availableMB‚ 1917.30078125% disk./.usageGB‚1260.8646392822266ÙÄ5 HÇ7‚(:½  ǵ‹Ó΃´ proc.cpu.threads‚17 disk.drbd2118.in‚0 disk.drbd1285.out‚0 disk.drbd2111.out‚0% disk./.usageGB‚1260.8646392822266 disk.nvme0n1.in‚ 123.63671875 disk.drbd1289.in‚0 disk.drbd2078.in‚0( proc.memory.availableMB‚ 1917.2421875 proc.memory.rssMB‚ 2914.3125 network.recv‚ 1.71113e+06 disk.sda.out‚ 933.73046875 disk.drbd1289.out‚0# memory_percent‚92.1987215677897& disk.nvme0n1.out‚57043.26904296875 network.sent‚ 2.3343025e+07 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0# disk.dm-0.out‚57043.26904296875( disk./.usagePercent‚70.5141511939273 disk.drbd1510.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.drbd1550.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1965.out‚0 cpu‚25.34140821502071 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.sda.in‚0.046875 disk.drbd1931.in‚0 disk.drbd2041.out‚0* proc.memory.percent‚11.858367919921875 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 120.2265625 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.046875 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 933.73046875 disk.drbd1356.in‚0 disk.drbd2041.in‚0&º‡-È7‚(:’ ǵ‹Óäðá³ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.566' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.85533333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽŽðaJÉ7‚(:¿  Öµ‹ÓÎ÷ñ³ disk.drbd2118.out‚0 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 proc.memory.rssMB‚ 2914.3125 disk.sda2.in‚0.046875 disk.drbd2111.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 123.63671875& disk.nvme0n1.out‚57043.40185546875 disk.drbd1510.in‚0 cpu‚25.341362277521714 disk.drbd1550.in‚0* proc.memory.percent‚11.858367919921875 disk.drbd1584.out‚0 disk.sda.in‚0.046875 disk.drbd1938.in‚0 disk.drbd2041.out‚0$ memory_percent‚92.19773610432944 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 network.sent‚ 2.3345305e+07( disk./.usagePercent‚70.5141511939273 disk.drbd1552.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.sda2.out‚ 933.98828125 disk.drbd1931.out‚0% disk./.usageGB‚1260.8646392822266 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 120.2890625 disk.sda1.in‚0 disk.drbd2118.in‚0 disk.sda.out‚ 933.98828125 disk.drbd1289.in‚0' proc.memory.availableMB‚ 1917.484375 disk.drbd1817.in‚0 network.recv‚ 1.711328e+06 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0# disk.dm-0.out‚57043.40185546875¼t7¬œÊ7‚(:‘ Öµ‹ÓŠûÖ³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚110.72433333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç€øË7‚(:’ 嵋Ó¡­À³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.603' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.86766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L¿óAÌ7‚(:¶  嵋Ó–«Ý³ disk.drbd2118.in‚0 disk.dm-0.out‚57043.78515625 disk.drbd2078.in‚0( disk./.usagePercent‚70.5141511939273 disk.sda.out‚ 934.421875 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0.046875 disk.drbd1584.in‚0 proc.cpu.threads‚17 disk.drbd2041.out‚0 disk.drbd1552.out‚0 disk.dm-0.in‚ 123.63671875 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1828.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0 disk.drbd1817.out‚0 disk.sda.in‚0.046875 disk.drbd1828.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 123.63671875 disk.drbd1510.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.19878514607748 disk.rbd1.out‚ 120.2890625 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0# disk.nvme0n1.out‚57043.78515625 cpu‚25.341316432387412 network.recv‚ 1.711436e+06( proc.memory.availableMB‚ 1917.2265625 network.sent‚ 2.3345413e+07 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0% disk./.usageGB‚1260.8646392822266 proc.memory.rssMB‚ 2914.3125 disk.drbd1356.in‚0* proc.memory.percent‚11.858367919921875 disk.sda2.out‚ 934.421875 disk.drbd1965.in‚0 disk.drbd1226.out‚0¤ôéÍ7‚(:’ ôµ‹Óñ¢Ô³ gpu.0.gpu‚97.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚268.904' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.63466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’åLÎ7‚(:Á  ôµ‹Ó³™ñ³ disk.drbd1817.out‚0 disk.sda2.in‚0.046875 disk.sda2.out‚ 934.9296875 disk.drbd2041.out‚0 disk.rbd1.out‚120.3212890625 disk.drbd1510.in‚0 disk.drbd2078.in‚0% disk.nvme0n1.out‚57044.1533203125 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 network.sent‚ 2.3345521e+07 network.recv‚ 1.711544e+06 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0) disk./.usagePercent‚70.51415098058946 disk.drbd1931.out‚0 disk.drbd2111.in‚0) proc.memory.availableMB‚ 1917.22265625 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0" disk.dm-0.out‚57044.1533203125 disk.drbd1965.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.nvme0n1.in‚ 133.69921875 disk.drbd1550.out‚0 disk.sda.in‚0.046875 disk.drbd1938.out‚0 disk.dm-0.in‚ 133.69921875$ memory_percent‚92.19880104064941 disk.drbd2041.in‚0% disk./.usageGB‚1260.8646354675293 disk.drbd1285.in‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 cpu‚25.341280258281326 disk.sda.out‚ 934.9296875 disk.drbd1828.out‚0* proc.memory.percent‚11.858367919921875 disk.drbd1584.in‚0 proc.memory.rssMB‚ 2914.3125 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0Ü /h4Ï7/ _wandbruntime‚24247  _runtime‚24247»Š¹ÊÐ7Ä & lr-AdamW/pg1‚6.494619232624752e-07 & lr-AdamW/pg2‚6.494619232624752e-07 trainer/global_step‚13349 % _timestamp‚1.7848634845168028e+09 _runtime‚24247.874042489 _step‚1111\¿lèÏÑ7É & lr-AdamW/pg1‚6.494619232624752e-07 & lr-AdamW/pg2‚6.494619232624752e-07 trainer/global_step‚13349 % _timestamp‚1.7848634845168028e+09 _runtime‚24247.874042489 _step‚1111ק¸¾Î4Ò7/ _wandbruntime‚24247  _runtime‚24247z¼ªú(Ó7˜P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3843878892148736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 434038272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3856272667481334 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13931121854464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3843878892148736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123959391 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40691052 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65947136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115907168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3842363501736592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76263679 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239866553 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35572632 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1068655761645568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115907162 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051622525792256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3857811385612800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13909140316774 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115907168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13932467290624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3857811385612800 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 _runtime‚24247.875322874 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 trainer/global_step‚13349 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3857810014003200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13932467290624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚404 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3842363527164560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0è[c° Ô7P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3857811385612800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13909140316774 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115907168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13932467290624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3857811385612800 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 _runtime‚24247.875322874 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 trainer/global_step‚13349 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3857810014003200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13932467290624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚404 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3842363527164560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3843878892148736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 434038272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3856272667481334 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13931121854464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3843878892148736 WGDeviceStatsMonitor.on_train_batch_start/allocatedÁ^þ×s_bytes.large_pool.peak‚ 46288510976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123959391 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40691052 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65947136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115907168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3842363501736592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76263679 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239866553 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35572632 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1068655761645568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115907162 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051622525792256Ø*Iâá4Õ7/ _wandbruntime‚24249  _runtime‚24249j€ï+('Ö7¢N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3842642303510368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239894086 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1068735152249344 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051700744067072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115915529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35575267 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40693998 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3856552490056862 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13933489805824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3844157827612672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13933489805824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123978551 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1068735579988992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13910161118526 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123968473 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239894086 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3842642328938336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13908816507128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3858091317418496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚24249.350349311 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115915529 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76269265 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13349 _step‚1113 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17034408182272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13932144362496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 % _timestamp‚1.7848634859927368e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115915535 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968Káˆø-'×7§N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123968473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40694396 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3858091317418496 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 % _timestamp‚1.7848634859927368e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115915535 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3842642303510368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239894086 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1068735152249344 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051700744067072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115915529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35575267 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40693998 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3856552490056862 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13933489805824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3844157827612672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13933489805824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123978551 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1068735579988992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13910161118526 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123968473 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239894086 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3842642328938336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13908816507128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3858091317418496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚24249.350349311 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115915529 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76269265 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13349 _step‚1113 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17034408182272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13932144362496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3857824356673536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚390 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055551844089856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13983656846848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚395 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3870262490013688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3857824382846976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3871806668084224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚13399 _runtime‚24334.512051009 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116325512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76544575 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76544180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40839943 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055551475998720 _step‚1116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3871808039693824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3871806668084224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13960244199582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13958899588296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35704627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3870263860052942 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 % _timestamp‚1.7848635711546657e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240750925 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3856303590425392 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124425407 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13982311410688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116325518 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124415343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35704632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116325512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1072646792450560’°]-#(ì7P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13960244199582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13958899588296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35704627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3870263860052942 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 % _timestamp‚1.7848635711546657e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240750925 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3856303590425392 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124425407 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13982311410688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116325518 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124415343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35704632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116325512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1072646792450560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13982311410688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17095374010368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17095316451840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116325518 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1072647218100224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3857824356673536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚390 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055551844089856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13983656846848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚395 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3870262490013688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3857824382846976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3871806668084224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚13399 _runtime‚24334.512051009 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116325512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76544575 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76544180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40839943 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055551475998720 _step‚1116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3871808039693824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3871806668084224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240768388 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1072726620024320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17096617308160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3856582392199168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13984679362048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40842568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13959920389936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3858103265964032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116333885 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3856582417627136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1072727047763968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚1117 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 % _timestamp‚1.7848635726262033e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35707267 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124424425 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚385 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76549835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116333879 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124424425 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3872086599882752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytescíí.small_pool.peak‚ 350627328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13983333918720 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3858103292137472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17096557659648 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3870543682628470 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3858103292137472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3858103265964032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240758304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124434503 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116333879 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 trainer/global_step‚13399_¥Ò†-'ï7§N YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1072726620024320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17096617308160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3856582392199168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13984679362048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40842568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13959920389936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3858103265964032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116333885 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3856582417627136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1072727047763968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 _step‚1117 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 % _timestamp‚1.7848635726262033e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35707267 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124424425 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚385 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76549835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116333879 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124424425 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3872086599882752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13983333918720 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3858103292137472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17096557659648 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3870543682628470 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3858103292137472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3858103265964032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240758304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124434503 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116333879 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 trainer/global_step‚13399 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3872087971499520 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13961265001334 _runtime‚24335.983805166 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240768388 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3872087971499520 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1055630062364672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13983333918720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240768388 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825ÝZÁ®ò4ð7/ _wandbruntime‚24336  _runtime‚24336c™V ñ7„ epoch‚47 0loss/FRt2/assignment_loss‚0.5876454571882884 1loss/FBt1/assignment_loss‚0.02532449646612119 1loss/SRqqt2/detection_loss‚0.5949341132503276 %loss/total_loss‚7.544946907181021 trainer/global_step‚13399 /loss/FBt2/detection_loss‚0.3836647815644723 0loss/FRt1/assignment_loss‚1.5888174970380284 2loss/FBt2/assignment_loss‚0.016095563523891555 /loss/FRt2/detection_loss‚1.3084156531478381 0loss/SRqqt1/detection_loss‚0.796641728130703 % _timestamp‚1.7848635726651385e+09 _runtime‚24336.022350189 _step‚1118 3loss/SRqqt1/assignment_loss‚0.48360712550042895 2loss/SRqqt2/assignment_loss‚0.4536488247418861 .loss/FRt1/detection_loss‚0.550066973458787 .loss/FBt1/detection_loss‚0.756084693170249$â}ýò7‰ /loss/FBt2/detection_loss‚0.3836647815644723 0loss/FRt1/assignment_loss‚1.5888174970380284 2loss/FBt2/assignment_loss‚0.016095563523891555 /loss/FRt2/detection_loss‚1.3084156531478381 0loss/SRqqt1/detection_loss‚0.796641728130703 % _timestamp‚1.7848635726651385e+09 _runtime‚24336.022350189 _step‚1118 3loss/SRqqt1/assignment_loss‚0.48360712550042895 2loss/SRqqt2/assignment_loss‚0.4536488247418861 .loss/FRt1/detection_loss‚0.550066973458787 .loss/FBt1/detection_loss‚0.756084693170249 epoch‚47 0loss/FRt2/assignment_loss‚0.5876454571882884 1loss/FBt1/assignment_loss‚0.02532449646612119 1loss/SRqqt2/detection_loss‚0.5949341132503276 %loss/total_loss‚7.544946907181021 trainer/global_step‚13399ÞÑ%ןLó7‚(:Á  ݶ‹ÓÔ …´ disk.drbd1356.in‚0 disk.drbd1552.out‚0 cpu‚25.34100984797838* proc.memory.percent‚11.858367919921875 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd2041.in‚0 network.sent‚ 2.3429135e+07 disk.drbd2078.in‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.drbd1226.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1226.out‚0 disk.sda.in‚0.046875) disk./.usagePercent‚70.48058547139948 disk.drbd2111.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚57092.21337890625 disk.drbd1550.in‚0 disk.sda2.out‚ 938.24609375# memory_percent‚92.1987533569336 disk.sda.out‚ 938.24609375% disk./.usageGB‚1260.2644500732422 disk.drbd2078.out‚0# disk.dm-0.out‚57092.21337890625 disk.drbd1965.in‚0 disk.nvme0n1.in‚ 138.12109375 disk.drbd1552.in‚0 disk.drbd1931.in‚0' proc.memory.availableMB‚ 1917.234375 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd1931.out‚0 proc.memory.rssMB‚ 2914.3125 disk.dm-0.in‚ 138.12109375 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda2.in‚0.046875 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚121.2509765625 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1510.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 network.recv‚ 1.720014e+06 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 Ì£ì‘ô7‚(:† ݶ‹Ó€Éá³ gpu.0.gpu‚26.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.354' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ḫ—õ7‚(:’ 춋ÓÕ‹¨³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚268.389' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.46300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 4tMö7‚(:  춋Ó¢Òʳ) disk./.usagePercent‚70.48058653808869 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 938.6484375 disk.nvme0n1.in‚ 138.12109375 disk.sda.in‚0.046875 disk.drbd1938.out‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 938.6484375 disk.drbd1289.out‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd1510.out‚0# disk.dm-0.out‚57092.34619140625 disk.drbd2111.in‚0 disk.drbd1828.out‚0 proc.cpu.threads‚17 network.sent‚ 2.3487319e+07 disk.drbd1226.out‚0 network.recv‚ 1.721938e+06 disk.sda2.in‚0.046875 disk.drbd2078.out‚0 disk.drbd1673.in‚0 disk.drbd1965.out‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1289.in‚0 proc.memory.rssMB‚ 2914.3125 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚57092.34619140625 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 121.283203125 disk.sda1.in‚0 disk.drbd1226.in‚0$ memory_percent‚92.19775199890137* proc.memory.percent‚11.858367919921875 cpu‚25.340965111880255% disk./.usageGB‚1260.2644691467285 disk.drbd1552.in‚0 disk.dm-0.in‚ 138.12109375 disk.drbd1510.in‚0) proc.memory.availableMB‚ 1917.48046875zu«Ù÷7‚(:’ û¶‹Ó¬ßà³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚287.197' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.73233333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zœ1Lø7‚(:Á  û¶‹Óœ¦´ disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1931.out‚0 disk.drbd1552.out‚0 disk.drbd1673.out‚0 disk.drbd1817.in‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd1356.out‚0& disk.nvme0n1.out‚57092.98193359375 disk.drbd1510.in‚0 disk.drbd1584.out‚0# memory_percent‚92.1987533569336 disk.drbd1285.out‚0 disk.sda.out‚ 938.98828125 proc.cpu.threads‚17) disk./.usagePercent‚70.48058547139948 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 938.98828125 disk.drbd1931.in‚0% disk./.usageGB‚1260.2644500732422 disk.drbd2041.in‚0 disk.drbd1285.in‚0 disk.sda.in‚0.046875 network.sent‚ 2.3492908e+07 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚ 121.384765625 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.046875 disk.drbd1552.in‚0' proc.memory.availableMB‚ 1917.234375 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 138.12109375 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0* proc.memory.percent‚11.858367919921875 cpu‚25.340959500586422 network.recv‚ 1.723002e+06 disk.drbd1938.out‚0# disk.dm-0.out‚57092.98193359375 disk.nvme0n1.in‚ 138.12109375 disk.drbd1510.out‚0 proc.memory.rssMB‚ 2914.3125HQaœù7‚(:‘ Š·‹ÓÁ£½³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚61.0 gpu.0.powerWatts‚108.8' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.266666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÄÕLú7‚(:Á  Š·‹ÓªîÚ³ disk.drbd1550.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0.046875 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd1356.out‚0 disk.drbd1510.out‚0& disk.nvme0n1.out‚57093.11083984375 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.046875 disk.drbd1552.in‚0 disk.drbd1828.in‚0# memory_percent‚92.1987533569336 disk.drbd1931.in‚0# disk.dm-0.out‚57093.11083984375 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.sda.out‚ 939.29296875 disk.sda2.out‚ 939.29296875 disk.drbd1938.in‚0 disk.drbd2111.out‚0 disk.drbd1584.out‚0 proc.memory.rssMB‚ 2914.3125 network.recv‚ 1.723954e+06 disk.sda1.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1226.in‚0' proc.memory.availableMB‚ 1917.234375) disk./.usagePercent‚70.48058547139948% disk./.usageGB‚1260.2644500732422 disk.drbd2111.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 138.12109375 cpu‚25.340904279009866 disk.drbd1550.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 138.12109375 disk.drbd1965.out‚0* proc.memory.percent‚11.858367919921875 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.rbd1.out‚ 121.447265625 disk.drbd1285.out‚0 disk.drbd2078.out‚0 network.sent‚ 2.3493886e+07gªVêû7‚(:„ ™·‹ÓîØʳ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚301.2' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚100.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ%mHü7‚(:½  ™·‹Óׄæ³ disk.dm-0.in‚ 143.78125 disk.drbd1965.in‚0 disk.drbd1510.in‚0( proc.memory.availableMB‚ 1917.3828125 disk.drbd2111.in‚0 disk.drbd1828.out‚0& disk.nvme0n1.out‚57093.48779296875 disk.drbd1584.out‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0% disk./.usageGB‚1260.2632942199707 disk.sda.in‚0.046875 disk.drbd1673.out‚0 network.recv‚ 1.724765e+06 disk.drbd2078.in‚0 disk.nvme0n1.in‚ 143.78125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1584.in‚0# disk.dm-0.out‚57093.48779296875 disk.drbd2118.out‚0 disk.sda2.in‚0.046875 network.sent‚ 2.3498907e+07 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.48052083003371 disk.drbd2078.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0$ memory_percent‚92.19814936319986 proc.memory.rssMB‚ 2914.3125 disk.sda2.out‚ 939.57421875 disk.drbd1550.out‚0 disk.sda1.in‚0 disk.sda.out‚ 939.57421875 cpu‚25.340858528194943 disk.drbd2041.in‚0 disk.rbd1.out‚ 121.447265625 disk.drbd1817.in‚0* proc.memory.percent‚11.858367919921875 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0ÃÃdª“ý7‚(:ˆ ¨·‹ÓàÇ«³ gpu.0.gpu‚98.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚303.945' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.315 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 îËIþ7‚(:¾  ¨·‹ÓÛëг disk.drbd1938.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.046875 disk.drbd1965.out‚0 disk.sda2.in‚0.046875 disk.drbd1931.in‚0) proc.memory.availableMB‚ 1909.88671875 disk.sda1.in‚0 disk.drbd1550.out‚0 disk.sda2.out‚ 939.828125 network.sent‚ 2.3501932e+07 network.recv‚ 1.725696e+06 disk.drbd1285.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.22865104675293 disk.drbd1226.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.drbd1584.in‚0 disk.drbd2118.out‚0 disk.drbd1510.out‚0 disk.drbd2111.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0# proc.memory.rssMB‚ 2921.55859375 cpu‚25.342244660100242 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 143.78125# disk.dm-0.out‚57093.63232421875 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚121.4794921875 disk.drbd1550.in‚0 disk.sda.out‚ 939.828125% disk./.usageGB‚1260.2642211914062 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0) proc.memory.percent‚11.88785235087077) disk./.usagePercent‚70.48057267112904& disk.nvme0n1.out‚57093.63232421875F)Èy4ÿ7/ _wandbruntime‚24424  _runtime‚24424"~bh€8Šv 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7763076424598694 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3583148419857025 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04385063797235489 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.269936740398407 *jet/accuracy_2_of_4‚0.8373957276344299 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032443489879369736 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08986023813486099 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22488434612751007 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31235456466674805 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.808417797088623 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 +jet/accuracy_1_of_1‚0.37284794449806213 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5870780348777771 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7493814826011658 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.690773606300354 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7211027145385742 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.794957160949707 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7551782727241516 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19170455634593964 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7050897479057312 +particle/sensitivity‚0.6422408819198608 &particle/f_score‚0.655168890953064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9224086999893188 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 /particle/accuracy_3_of_3‚0.9993678331375122 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6581174731254578 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17572377622127533 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8026137948036194 +particle/specificity‚0.9130682349205017 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22842039167881012 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 *jet/accuracy_1_of_3‚0.9380978345870972 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6537144780158997 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5164191722869873 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3878452777862549 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07369927316904068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26952433586120605 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4075712263584137 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3097807466983795 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34962227940559387 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6435168981552124 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_2_of_3‚0.6062600016593933 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3695864677429199 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3583148419857025 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029224084690213203 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.27160272002220154 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6769906282424927 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.881477952003479 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7160285711288452 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3824477195739746 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3585486114025116 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6625214219093323 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6307708621025085 :"Purity/*FRt*SRqqt*FBt/e?ö¶¨vent_purity‚0.38848111033439636 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35586950182914734 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5266430974006653 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30087122321128845 7validation_average_jet_accuracy‚0.40680667757987976 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17971079051494598 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6398096084594727 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07885482162237167 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6921738982200623 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 (particle/accuracy‚0.8549630045890808 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚47 trainer/global_step‚13439 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _step‚1119 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.784863661467413e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2790977358818054 *jet/accuracy_1_of_4‚0.9832039475440979 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3852396011352539 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4590897262096405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6876805424690247 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7694299817085266 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.691164493560791 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03461159020662308 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 *jet/accuracy_2_of_2‚0.1729692667722702 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6382198333740234 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17894676327705383 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053134314715862274 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6640806794166565 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2815077006816864 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2974303364753723 7"Purity/0FRt*SRqqt1FBt/event_purity‚0.87553870677948 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7198843955993652 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 /particle/accuracy_4_of_4‚0.9640940427780151 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.633392333984375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37189558148384094 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7210091948509216 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6435168981552124 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027649154886603355 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5481786727905273 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.37034517526626587 particle/accuracy_1_of_1‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14848235249519348 _runtime‚24424.825496501 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7942200303077698 *jet/accuracy_1_of_2‚0.7909737825393677 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7186370491981506 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1634388267993927 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7815518379211426 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2892943322658539 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6322174072265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 4 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3599833548069 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6980623602867126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 )loss/val_total_loss‚7.713819552109311 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096lNC;8v 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07369927316904068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26952433586120605 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4075712263584137 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3097807466983795 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34962227940559387 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6435168981552124 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 *jet/accuracy_2_of_3‚0.6062600016593933 ,jet/accuracy_4_of_4‚0.018756305798888206 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3695864677429199 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3583148419857025 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029224084690213203 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.27160272002220154 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6769906282424927 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.881477952003479 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7160285711288452 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3824477195739746 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3585486114025116 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6625214219093323 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6307708621025085 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38848111033439636 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35586950182914734 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5266430974006653 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30087122321128845 7validation_average_jet_accuracy‚0.40680667757987976 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17971079051494598 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6398096084594727 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07885482162237167 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6921738982200623 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 (particle/accuracy‚0.8549630045890808 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚47 trainer/global_step‚13439 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _step‚1119 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.784863661467413e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2790977358818054 *jet/accuracy_1_of_4‚0.9832039475440979 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3852396011352539 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4590897262096405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6876805424690247 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7694299817085266 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.691164493560791 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03461159020662308 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 *jet/accuracy_2_of_2‚0.1729692667722702 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6382198333740234 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17894676327705383 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053134314715862274 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6640806794166565 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2815077006816864 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2974303364753723 7"Purity/0FRt*SRqqt1FBt/event_purity‚0.87553870677948 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7198843955993652 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 /particle/accuracy_4_of_4‚0.9640940427780151 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.633392333984375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37189558148384094 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7210091948509216 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6435168981552124 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027649154886603355 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5481786727905273 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.37034517526626587 particle/accuracy_1_of_1‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14848235249519348 _runtime‚24424.825496501 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7942200303077698 *jet/accuracy_1_of_2‚0.7909737825393677 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7186370491981506 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1634388267993927 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7815518379211426 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2892943322658539 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6322174072265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 4 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3599833548069 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6980623602867126 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 )loss/val_total_loss‚7.713819552109311 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7763076424598694 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3583148419857025 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04385063797235489 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.269936740398407 *jet/accuracy_2_of_4‚0.8373957276344299 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032443489879369736 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08986023813486099 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22488434612751007 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31235456466674805 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.808417797088623 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 +jet/accuracy_1_of_1‚0.37284794449806213 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5870780348777771 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7493814826011658 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.690773606300354 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7211027145385742 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.794957160949707 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7551782727241516 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19170455634593964 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7050897479057312 +particle/sensitivity‚0.6422408819198608 &particle/f_score‚0.655168890953064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9224086999893188 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 /particle/accuracy_3_of_3‚0.9993678331375122 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6581174731254578 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17572377622127533 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8026137948036194 +particle/specificity‚0.9130682349205017 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22842039167881012 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 *jet/accuracy_1_of_3‚0.9380978345870972 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6537144780158997 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5164191722869873 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3878452777862549ß±É`ß‚8‚(:’ ··‹Ó§ÑÞ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚282.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.16566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QèMƒ8‚(:  ··‹Ó„Èþ³ disk.drbd2041.in‚0 disk.sda2.out‚ 940.15234375 disk.drbd1552.in‚0 disk.sda2.in‚0.046875 disk.drbd1931.out‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1908.62890625 disk.drbd1552.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚57093.98681640625 cpu‚25.34276971630211 disk.sda1.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.sda.in‚0.046875 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 network.recv‚ 1.726649e+06 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1550.in‚0* proc.memory.percent‚11.890554428100586 disk.drbd2118.out‚0 disk.rbd1.out‚121.5810546875 disk.drbd1817.in‚0 disk.drbd1938.out‚0) disk./.usagePercent‚70.48062557891357 disk.drbd1673.out‚0 disk.dm-0.in‚ 143.78125 disk.drbd1510.in‚0 network.sent‚ 2.3504919e+07 disk.sda.out‚ 940.15234375 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0# disk.dm-0.out‚57093.98681640625 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0$ memory_percent‚92.23376909891763# proc.memory.rssMB‚ 2922.22265625 proc.cpu.threads‚17% disk./.usageGB‚1260.2651672363281 disk.drbd1584.in‚0q“4„8/ _wandbruntime‚24440  _runtime‚24440ZótÊ…8Ä & lr-AdamW/pg1‚4.655036317540051e-07 & lr-AdamW/pg2‚4.655036317540051e-07 trainer/global_step‚13449 % _timestamp‚1.7848636775866685e+09 _runtime‚24440.944895407 _step‚1120ž@ûφ8É % _timestamp‚1.7848636775866685e+09 _runtime‚24440.944895407 _step‚1120 & lr-AdamW/pg1‚4.655036317540051e-07 & lr-AdamW/pg2‚4.655036317540051e-07 trainer/global_step‚13449à… ¢4‡8/ _wandbruntime‚24440  _runtime‚24440±[ˆ8˜P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76857116 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3874330497923072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3888370988936192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116833294 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14040464839680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35843524 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17164103307264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14040464839680 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1076693862626816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3888370988936192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241751896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124918602 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3874330524096512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 _runtime‚24440.944976226 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14016956921699 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14039119403520 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41013339 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 trainer/global_step‚13449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3888369617326592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35843519 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76856858 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124928666 ]HDeviceStatsMonitor.on_traÞÒk in_batch_start/allocated_bytes.large_pool.freed‚3874330497923072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14039119403520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3874330524096512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241761966 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1059529793809408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41013592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 _step‚1121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116833300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848636775873868e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124918602 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116833294 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124928666 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116833300 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1076694265207808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17164068817408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3888369617326592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¬þ-Ç#(‰8P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3888370988936192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116833294 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14040464839680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35843524 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17164103307264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14040464839680 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1076693862626816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3888370988936192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241751896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124918602 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3874330524096512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 _runtime‚24440.944976226 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14016956921699 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14039119403520 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41013339 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 trainer/global_step‚13449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3888369617326592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35843519 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76856858 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124928666 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3874330497923072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14039119403520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3874330524096512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241761966 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1059529793809408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41013592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 _step‚1121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116833300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 % _timestamp‚1.7848636775873868e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124918602 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116833294 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124928666 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116833300 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1076694265207808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17164068817408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3888369617326592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76857116 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3874330497923072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34489856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976áTÛ ¯4Š8/ _wandbruntime‚24442  _runtime‚24442>4ØŽ&'‹8 N RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241779429 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14040141911552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3887098115151189 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3888650920741888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14040141911552 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241779429 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124927684 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1076774168042496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14016633112053 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124927684 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3874609433387008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116841667 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1076773765468672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241769345 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76862508 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1059608748266496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1059608380175360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚1122 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35846159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3888650920741888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3874609407213568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17165385293312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14041487354880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124937762 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41016603 trainer/global_step‚13449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14041487354880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3887099485190555 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124937762 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3874609433387008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚410163499¬¾[$Œ8¥N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116841667 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1076773765468672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241769345 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76862508 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1059608748266496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1059608380175360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 _step‚1122 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35846159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3888650920741888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3874609407213568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17165385293312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14041487354880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124937762 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41016603 trainer/global_step‚13449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14041487354880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3887099485190555 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124937762 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3874609433387008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41016349 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241779429 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14040141911552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3887098115151189 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3888650920741888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14040141911552 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241779429 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124927684 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1076774168042496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402573824 YFDevice{­\/ÐStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14016633112053 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124927684 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3874609433387008ân &´48/ _wandbruntime‚24442  _runtime‚24442çXóŽ8Š 0loss/FRt2/assignment_loss‚0.5788746570007764 /loss/FRt1/detection_loss‚0.5233228073036997 trainer/global_step‚13449 0loss/FRt1/assignment_loss‚1.6496626203297637 3loss/SRqqt2/assignment_loss‚0.45608016902896337 3loss/FBt2/assignment_loss‚0.0030421689205795133 /loss/FRt2/detection_loss‚1.4137812023734604 1loss/SRqqt2/detection_loss‚0.5659167227367876 % _timestamp‚1.7848636790991473e+09 /loss/FBt1/detection_loss‚0.6686031906643793 %loss/total_loss‚7.402928621659827 2loss/SRqqt1/assignment_loss‚0.4784636716795141 3loss/FBt1/assignment_loss‚0.0011768695447974171 0loss/SRqqt1/detection_loss‚0.864271803680707 0loss/FBt2/detection_loss‚0.19973273839639813 epoch‚48 _runtime‚24442.456339459 _step‚1123M…K;8 epoch‚48 _runtime‚24442.456339459 _step‚1123 0loss/FRt2/assignment_loss‚0.5788746570007764 /loss/FRt1/detection_loss‚0.5233228073036997 trainer/global_step‚13449 0loss/FRt1/assignment_loss‚1.6496626203297637 3loss/SRqqt2/assignment_loss‚0.45608016902896337 3loss/FBt2/assignment_loss‚0.0030421689205795133 /loss/FRt2/detection_loss‚1.4137812023734604 1loss/SRqqt2/detection_loss‚0.5659167227367876 % _timestamp‚1.7848636790991473e+09 /loss/FBt1/detection_loss‚0.6686031906643793 %loss/total_loss‚7.402928621659827 2loss/SRqqt1/assignment_loss‚0.4784636716795141 3loss/FBt1/assignment_loss‚0.0011768695447974171 0loss/SRqqt1/detection_loss‚0.864271803680707 0loss/FBt2/detection_loss‚0.19973273839639813ãëu ´8‚(:’ Æ·‹ÓÉì²³ gpu.0.gpu‚98.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚299.732' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚99.91066666666669 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜»÷§@‘8‚(:µ  Æ·‹ÓãÓ³ disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1226.in‚0 disk.drbd2078.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.sda.out‚940.5 disk.sda2.in‚0.046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.48062237884595 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2922.22265625 cpu‚25.342734020329683 disk.drbd1828.out‚0 network.sent‚ 2.3556772e+07 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1260.2651100158691 disk.dm-0.in‚ 143.78125# disk.dm-0.out‚57094.17822265625 proc.cpu.threads‚17 disk.drbd2111.in‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚57094.17822265625* proc.memory.percent‚11.890554428100586 network.recv‚ 1.728527e+06 disk.sda2.out‚940.5 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1965.out‚0 disk.drbd2041.in‚0$ memory_percent‚92.23148028055826) proc.memory.availableMB‚ 1909.19140625 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.rbd1.out‚121.6435546875 disk.drbd1356.in‚0 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd1510.out‚0 disk.sda.in‚0.046875 disk.drbd1673.out‚0 disk.drbd1938.out‚0 disk.drbd1510.in‚0çùIN’8‚(:à  Õ·‹Ó£¡Ó³) disk./.usagePercent‚70.48072926110422 disk.drbd1817.out‚0 disk.drbd1673.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.890554428100586 disk.drbd2118.out‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0& disk.nvme0n1.out‚57094.59814453125 disk.dm-0.in‚ 143.78125 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 940.81640625 disk.drbd1550.in‚0% disk./.usageGB‚1260.2670211791992 disk.drbd1289.out‚0 cpu‚25.342677254325356 network.sent‚ 2.3618925e+07 disk.drbd2118.in‚0$ memory_percent‚92.23202069600424 disk.sda.in‚0.046875 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd1938.in‚0 disk.rbd1.out‚121.6435546875 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1285.in‚0 disk.drbd1226.in‚0 disk.sda2.out‚ 940.81640625 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2922.22265625 proc.cpu.threads‚17# disk.dm-0.out‚57094.59814453125 disk.drbd2041.in‚0 network.recv‚ 1.730654e+06 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0 disk.sda2.in‚0.046875 disk.drbd1938.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1909.05859375îÉRž“8‚(:“ Õ·‹Óž±³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚61.0 gpu.0.powerWatts‚185.018' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚61.672666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09 +öž”8‚(:“ ä·‹Óà„¾³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚302.454' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.81800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fqB•8‚(:·  ä·‹Ó’×ã³ disk.drbd1931.out‚0) disk./.usagePercent‚70.48072798107718 disk.drbd1552.out‚0 disk.sda.in‚0.046875 disk.drbd1673.in‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.2669982910156 disk.sda1.in‚0 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd2078.in‚0* proc.memory.percent‚11.890554428100586 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 143.78125 proc.cpu.threads‚17 network.sent‚ 2.3621754e+07 disk.drbd1510.out‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2922.22265625 disk.sda1.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.23237037658691 disk.sda.out‚941.125 disk.drbd1673.out‚0 disk.drbd2111.out‚0 disk.drbd1931.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 143.78125 disk.sda2.in‚0.046875 disk.drbd2078.out‚0 disk.rbd1.out‚ 121.67578125 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1817.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0# disk.dm-0.out‚57094.81689453125) proc.memory.availableMB‚ 1908.97265625 network.recv‚ 1.730852e+06 disk.sda2.out‚941.125 disk.drbd1938.out‚0& disk.nvme0n1.out‚57094.81689453125 cpu‚25.342650859762486 disk.drbd1965.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0Sž`Ê–8‚(:’ ó·‹ÓÜݳ gpu.0.gpu‚81.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚295.834' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.61133333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â6¬šC—8‚(:¸  ó·‹Ó¯Õ‚´ disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1673.in‚0# disk.dm-0.out‚57095.20654296875 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.drbd1817.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚941.5625 disk.drbd1965.out‚0 disk.drbd1285.in‚0 network.recv‚ 1.730918e+06 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0$ memory_percent‚92.23310152689615 disk.sda1.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.046875 disk.drbd2041.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 143.78125* proc.memory.percent‚11.890554428100586 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0 cpu‚25.342614070932168 disk.drbd2118.in‚0 disk.sda2.in‚0.046875 disk.drbd1938.in‚0& disk.nvme0n1.out‚57095.20654296875 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.sda.out‚941.5625 disk.drbd1965.in‚0 disk.drbd2041.out‚0) proc.memory.availableMB‚ 1908.79296875# proc.memory.rssMB‚ 2922.22265625% disk./.usageGB‚1260.2669982910156 disk.dm-0.in‚ 143.78125 disk.rbd1.out‚ 121.77734375) disk./.usagePercent‚70.48072798107718 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd2111.in‚0 disk.rbd1.in‚0 disk.drbd1510.out‚0 network.sent‚ 2.362182e+07 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1289.out‚0¸`M”I˜8‚(:¾  ‚¸‹Óë‘ý³ disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1510.out‚0 disk.drbd2041.in‚0 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.sda1.in‚0 disk.sda.in‚0.046875 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1938.out‚0 disk.drbd1226.out‚0) proc.memory.availableMB‚ 1908.59765625 disk.nvme0n1.in‚ 143.78125 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚57095.73779296875 disk.drbd1828.in‚0& disk.nvme0n1.out‚57095.73779296875 network.recv‚ 1.730984e+06) disk./.usagePercent‚70.48072798107718 disk.dm-0.in‚ 143.78125 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 cpu‚25.34255756258878 network.sent‚ 2.3621886e+07 disk.drbd1965.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1931.out‚0 disk.rbd1.out‚ 121.83984375 disk.drbd1356.out‚0* proc.memory.percent‚11.890554428100586 disk.drbd2118.in‚0% disk./.usageGB‚1260.2669982910156 disk.drbd1673.out‚0 disk.sda2.in‚0.046875 disk.drbd2111.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0$ memory_percent‚92.23389625549316# proc.memory.rssMB‚ 2922.22265625 disk.sda2.out‚ 941.8828125 disk.drbd1510.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.sda.out‚ 941.8828125—f y™8‚(:’ ‚¸‹Óœ®Ò³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚282.669' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.22299999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0· Ë|š8‚(:’ ‘¸‹Óä¸Ý³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚278.219' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.73966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x‰[âG›8‚(:¼  ‘¸‹Óµ”ý³ disk.drbd2111.in‚0% disk./.usageGB‚1260.2669982910156 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 121.83984375 disk.drbd1289.in‚0( proc.memory.availableMB‚ 1908.4921875" disk.dm-0.out‚57096.0654296875 disk.rbd1.in‚0% disk.nvme0n1.out‚57096.0654296875 disk.drbd2118.in‚0 disk.drbd1817.out‚0 proc.cpu.threads‚17 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 network.sent‚ 2.3630854e+07 disk.drbd1817.in‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1673.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0* proc.memory.percent‚11.890554428100586 disk.dm-0.in‚ 143.78125 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 cpu‚25.342531556381104 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.drbd1552.in‚0 disk.drbd1938.out‚0# proc.memory.rssMB‚ 2922.22265625) disk./.usagePercent‚70.48072798107718 disk.sda2.in‚0.046875 disk.drbd2041.in‚0$ memory_percent‚92.23432540893555 disk.drbd1965.out‚0 disk.drbd1550.out‚0 network.recv‚ 1.733889e+06 disk.drbd2118.out‚0 disk.sda.out‚ 942.4765625 disk.drbd1931.out‚0 disk.drbd1584.out‚0 disk.sda.in‚0.046875 disk.sda2.out‚ 942.4765625 disk.drbd1965.in‚0I÷¿(4œ8/ _wandbruntime‚24527  _runtime‚24527€-mÊ8Ä % _timestamp‚1.7848637641238236e+09 _runtime‚24527.481060295 _step‚1124 & lr-AdamW/pg1‚3.849418802241791e-07 & lr-AdamW/pg2‚3.849418802241791e-07 trainer/global_step‚13499¹~ÍÏž8É _runtime‚24527.481060295 _step‚1124 & lr-AdamW/pg1‚3.849418802241791e-07 & lr-AdamW/pg2‚3.849418802241791e-07 trainer/global_step‚13499 % _timestamp‚1.7848637641238236e+09ä½å­4Ÿ8/ _wandbruntime‚24527  _runtime‚24527° L†( 8–P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117251650 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3902367643017216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14090308959744 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3902367643017216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35975519 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3886742768954160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14091654395904 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3900809485147381 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14090308959744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77138113 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 # _timestamp‚1.78486376412489e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14091654395904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1125 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117251644 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 _runtime‚24527.482436624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117251644 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3888275988621312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242626198 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242636268 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3886742794382128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117251650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77137778 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1063459112107008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41162589 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41162259 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1063459480198144 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14068060804507 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125384618 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3902366271407616 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3888275988621312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚335 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17227385221120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3900810855186635 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14066716193221 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3888275962447872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125374554 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1080686497328128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5y¦ËÍ¡8›P aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3886742794382128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117251650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77137778 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1063459112107008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41162589 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41162259 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1063459480198144 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14068060804507 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125384618 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3902366271407616 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3888275988621312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚335 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17227385221120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3900810855186635 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14066716193221 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3888275962447872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125374554 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1080686497328128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117251650 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3902367643017216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14090308959744 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3902367643017216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35975519 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3886742768954160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14091654395904 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3900809485147381 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14090308959744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77138113 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 # _timestamp‚1.78486376412489e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14091654395904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1125 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117251644 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 _runtime‚24527.482436624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117251644 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3888275988621312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242626198 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242636268åÇ^ü4¢8/ _wandbruntime‚24528  _runtime‚245281ä·''£8¡N WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14092676911104 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117260011 $ _timestamp‚1.784863765601116e+09 _step‚1126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125393714 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125383636 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242653731 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242653731 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125383636 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14092676911104 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1063538066564096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _runtime‚24528.958759379 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242643647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77143365 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1080766339091456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3887021596155904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17228687684096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242643647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14067736994861 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1080766754248192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17228640618496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117260017 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41165535 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3901090677762163 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117260011 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 trainer/global_step‚13499 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚329 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3887021570727936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3902647574822912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3902647574822912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3888554897911808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14091331467776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1063537698472960 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3888554897911808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3902646203206144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Hù,'¤8¦N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1080766754248192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17228640618496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117260017 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41165535 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3901090677762163 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117260011 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 trainer/global_step‚13499 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚329 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 415156736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3887021570727936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3902647574822912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3902647574822912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3888554897911808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14091331467776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1063537698472960 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3888554897911808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3902646203206144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14092676911104 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117260011 $ _timestamp‚1.784863765601116e+09 _step‚1126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125393714 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125383636 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242653731 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242653731 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125383636 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14092676911104 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1063538066564096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 _runtime‚24528.958759379 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47065600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242643647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77143365 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1080766339091456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3887021596155904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17228687684096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242643647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14067736994861æµuç4¥8/ _wandbruntime‚24528  _runtime‚24528–ôM|¦8ˆ 1loss/FBt2/assignment_loss‚0.03386103099710229 /loss/FRt2/detection_loss‚1.3275802056162145 1loss/SRqqt2/detection_loss‚0.5918184039282482 0loss/FBt2/detection_loss‚0.47859534318826996 %loss/total_loss‚7.216577896366113 trainer/global_step‚13499 $ _timestamp‚1.784863765637322e+09 _step‚1127 3loss/SRqqt1/assignment_loss‚0.44213803432706944 1loss/SRqqt1/detection_loss‚0.8305883157202193 epoch‚48 2loss/SRqqt2/assignment_loss‚0.4499134305425814 3loss/FBt1/assignment_loss‚6.100466299704029e-05 /loss/FRt1/detection_loss‚0.5068939746882163 0loss/FRt1/assignment_loss‚1.5389407998546742 0loss/FRt2/assignment_loss‚0.6288320095392695 /loss/FBt1/detection_loss‚0.3873553433012502 _runtime‚24528.994531177`仧8 1loss/FBt2/assignment_loss‚0.03386103099710229 /loss/FRt2/detection_loss‚1.3275802056162145 1loss/SRqqt2/detection_loss‚0.5918184039282482 0loss/FBt2/detection_loss‚0.47859534318826996 %loss/total_loss‚7.216577896366113 trainer/global_step‚13499 $ _timestamp‚1.784863765637322e+09 _step‚1127 3loss/SRqqt1/assignment_loss‚0.44213803432706944 1loss/SRqqt1/detection_loss‚0.8305883157202193 epoch‚48 2loss/SRqqt2/assignment_loss‚0.4499134305425814 3loss/FBt1/assignment_loss‚6.100466299704029e-05 /loss/FRt1/detection_loss‚0.5068939746882163 0loss/FRt1/assignment_loss‚1.5389407998546742 0loss/FRt2/assignment_loss‚0.6288320095392695 /loss/FBt1/detection_loss‚0.3873553433012502 _runtime‚24528.994531177ç–™õÌž¨8‚(:“  ¸‹Ó«Î³ gpu.0.gpu‚97.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.512' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.17066666666668 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B”÷ÒF©8‚(:»   ¸‹Ó¼óâ³ disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1817.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 942.84375 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1938.out‚0 disk.drbd1938.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1828.out‚0# memory_percent‚92.2345002492269 disk.drbd1285.in‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.dm-0.in‚ 143.78125 disk.drbd2111.out‚0 disk.drbd1828.in‚0& disk.nvme0n1.out‚57096.59130859375 network.recv‚ 1.734021e+06$ disk./.usageGB‚1260.267967224121 disk.drbd1965.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.890554428100586 disk.nvme0n1.in‚ 143.78125 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1908.44921875# proc.memory.rssMB‚ 2922.22265625 network.sent‚ 2.3631658e+07 disk.sda1.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1552.in‚0 disk.rbd1.in‚0 cpu‚25.342495868125813) disk./.usagePercent‚70.48078216888875 disk.sda2.in‚0.046875 disk.drbd1931.out‚0# disk.dm-0.out‚57096.59130859375 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 942.84375 disk.rbd1.out‚121.8720703125 disk.drbd1226.in‚0 proc.cpu.threads‚17 disk.sda.in‚0.046875 disk.drbd1673.out‚0 disk.drbd1965.out‚0 disk.drbd1584.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0©“¬Mª8‚(:  ¯¸‹Óµîñ³ disk.drbd2041.out‚0 disk.nvme0n1.in‚ 143.78125 disk.sda1.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0* proc.memory.percent‚11.890554428100586 disk.drbd2118.out‚0 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd1965.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚122.1533203125( proc.memory.availableMB‚ 1908.4453125 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd1226.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1552.out‚0 disk.sda.out‚ 943.23046875 disk.drbd1356.out‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.2699127197266 disk.sda1.in‚0 disk.drbd2041.in‚0 disk.drbd2111.out‚0& disk.nvme0n1.out‚57097.42529296875 disk.drbd1510.out‚0 cpu‚25.342439189411095 disk.sda2.out‚ 943.23046875 disk.drbd2078.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.046875) disk./.usagePercent‚70.48089097118758 disk.drbd1817.out‚0 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1510.in‚0# proc.memory.rssMB‚ 2922.22265625 network.recv‚ 1.734087e+06 network.sent‚ 2.3631724e+07 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0$ memory_percent‚92.23451614379883 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 143.78125# disk.dm-0.out‚57097.42529296875 proc.cpu.threads‚17 disk.sda.in‚0.046875 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0ÿb„ª«8‚(:’ ¯¸‹Óíæ¡Ñ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.825' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.94166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Ã±Ù¬8‚(:’ ¾¸‹Ó‡ý׳ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚181.069' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.35633333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uôJ­8‚(:¿  ¾¸‹Ó›êô³ cpu‚25.342403105456263 disk.sda2.in‚0.046875 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 143.78125* proc.memory.percent‚11.890554428100586 network.sent‚ 2.3696918e+07 disk.drbd2041.out‚0 disk.sda1.out‚0 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd1828.in‚0 disk.drbd1289.out‚0# disk.dm-0.out‚57097.75537109375 disk.drbd1817.in‚0 disk.drbd1285.in‚0 network.recv‚ 1.737111e+06% disk./.usageGB‚1260.2699546813965 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 143.78125$ memory_percent‚92.23453203837076 proc.cpu.threads‚17 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0) disk./.usagePercent‚70.48089331790382 disk.drbd1289.in‚0 disk.drbd1938.out‚0 disk.rbd1.out‚122.2158203125 disk.drbd1550.in‚0 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚57097.75537109375 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.sda2.out‚ 943.671875 disk.sda1.in‚0 disk.sda.in‚0.046875 disk.sda.out‚ 943.671875 disk.drbd2041.in‚0 disk.drbd1673.in‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1550.out‚0) proc.memory.availableMB‚ 1908.44140625“ýìäž®8‚(:“ ͸‹ÓÀÜß³ gpu.0.gpu‚39.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚300.394' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.13133333333333 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I:«¯8‚(:·  ͸‹Óª‚ÿ³ disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0$ memory_percent‚92.23451614379883 disk.rbd1.in‚0 disk.sda2.in‚0.046875 disk.drbd1938.in‚0 disk.drbd1938.out‚0" disk.nvme0n1.out‚ 57098.4765625 cpu‚25.342366937649647$ disk./.usageGB‚1260.269931793213 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.rbd1.out‚122.2158203125 disk.drbd1550.out‚0 disk.sda1.out‚0 disk.sda.out‚ 943.9921875 disk.drbd1356.out‚0 disk.sda2.out‚ 943.9921875 disk.nvme0n1.in‚ 143.78125 disk.drbd1550.in‚0# proc.memory.rssMB‚ 2922.22265625 disk.sda.in‚0.046875 disk.drbd1673.in‚0 disk.dm-0.out‚ 57098.4765625 disk.drbd2078.out‚0 disk.drbd1584.out‚0( proc.memory.availableMB‚ 1908.4453125 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0* proc.memory.percent‚11.890554428100586 network.recv‚ 1.737177e+06) disk./.usagePercent‚70.48089203787679 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.dm-0.in‚ 143.78125 disk.drbd2078.in‚0 disk.drbd1552.in‚0 proc.cpu.threads‚17 network.sent‚ 2.3696984e+07 disk.drbd1289.in‚0 disk.drbd2041.out‚0 disk.drbd1828.in‚0 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1931.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1356.in‚0 ±fšA.disk.drbd1289.out‚0 disk.drbd1285.out‚0c&뜰8‚(:‘ ܸ‹Óûê³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚275.16' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.72000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÄB±8‚(:·  ܸ‹Ó‹ë„´ disk.drbd1226.in‚0 disk.dm-0.in‚ 143.78125 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 cpu‚25.342330897138464 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.890554428100586 disk.drbd2111.in‚0# disk.nvme0n1.out‚57098.66015625 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.drbd1673.out‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0) disk./.usagePercent‚70.48089203787679 network.sent‚ 2.369705e+07 disk.drbd1828.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1965.in‚0$ memory_percent‚92.23464330037436$ disk./.usageGB‚1260.269931793213 disk.drbd1285.out‚0 disk.sda1.in‚0 disk.drbd1356.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0 disk.sda2.out‚ 944.3515625 disk.dm-0.out‚57098.66015625 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.sda2.in‚0.046875( proc.memory.availableMB‚ 1908.4140625 disk.drbd1584.in‚0 disk.sda.out‚ 944.3515625 disk.drbd1673.in‚0 disk.rbd1.out‚ 122.427734375 disk.nvme0n1.in‚ 143.78125 disk.drbd1510.out‚0 disk.sda.in‚0.046875 network.recv‚ 1.737243e+06 disk.rbd1.in‚0uþá±4²8/ _wandbruntime‚24614  _runtime‚24614}Ù÷?ɳ8à & lr-AdamW/pg1‚3.120061373140607e-07 & lr-AdamW/pg2‚3.120061373140607e-07 trainer/global_step‚13549 $ _timestamp‚1.784863851086618e+09 _runtime‚24614.443864482 _step‚1128öG.δ8È _step‚1128 & lr-AdamW/pg1‚3.120061373140607e-07 & lr-AdamW/pg2‚3.120061373140607e-07 trainer/global_step‚13549 $ _timestamp‚1.784863851086618e+09 _runtime‚24614.443864482èO.ʪ4µ8/ _wandbruntime‚24614  _runtime‚24614Eb2(¶8—P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41310210 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117669994 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3902221453146112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3916362925488640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117670000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3916362925488640 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1067388430404608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17289741017600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3900682883070928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3902221426972672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117669994 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125830506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067388798495744 % _timestamp‚1.7848638510876558e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3914802047758243 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3900682857642960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3902221453146112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125840570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3916364297098240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125830506 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3902221426972672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14142843952128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243510570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77417729 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14117820076029 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0&[,L"(·8œP G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3902221453146112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125840570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3916364297098240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125830506 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3902221426972672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14142843952128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243510570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77417729 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14117820076029 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41310210 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117669994 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3902221453146112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3916362925488640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117670000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3916362925488640 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1067388430404608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17289741017600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3900682883070928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3902221426972672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117669994 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125830506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚356 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067388798495744 % _timestamp‚1.7848638510876558e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚361 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3914802047758243 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3900682857642960齙ϸ8‚(:’ 븋Ó°…Ò³ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚176.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚58.87933333333334 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05DwrB¹8‚(:·  븋Óljð³$ disk./.usageGB‚1260.269931793213 disk.drbd2111.in‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd1285.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 network.recv‚ 1.738275e+06 disk.rbd1.out‚ 122.529296875 disk.drbd1226.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1550.in‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 proc.cpu.threads‚17 disk.drbd1828.in‚0 disk.drbd1673.in‚0 disk.drbd2041.in‚0 network.sent‚ 2.3698108e+07 disk.drbd1226.out‚0 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.sda2.out‚ 944.7265625* proc.memory.percent‚11.890554428100586) disk./.usagePercent‚70.48089203787679 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1584.out‚0 cpu‚25.342305482306458 disk.drbd1510.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 944.7265625# proc.memory.rssMB‚ 2922.22265625 disk.drbd1356.in‚0 disk.drbd1584.in‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0 disk.dm-0.out‚57098.99609375 disk.drbd1965.in‚0 disk.drbd1285.out‚0# disk.nvme0n1.out‚57098.99609375 disk.dm-0.in‚ 143.78125 disk.sda.in‚0.046875 disk.drbd1938.in‚0$ memory_percent‚92.23461151123047 disk.sda1.in‚0 disk.sda2.in‚0.046875' proc.memory.availableMB‚ 1908.421875 Î~¾4º8/ _wandbruntime‚24615  _runtime‚24615˜!lJH»8¢N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3900961659416736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14142521024000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1084757992361472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3902500336263168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14120185489067 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1084758420101120 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067467016770560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3915080500294405 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3900961684844704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14118840877669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36110159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848638525633874e+09 _step‚1130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3916644228903936 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77423270 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚24615.920990678 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17291035239424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3916642857287168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 243528033 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117678367 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3916644228903936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41313468 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117678361 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125849666 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14142521024000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 FDeviceStatsMonitor.VFÍ"à on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243528033 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117678361 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14143866467328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14143866467328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17290975590912 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243528033 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117678361 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14143866467328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14143866467328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17290975590912 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1084757992361472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3902500336263168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14120185489067 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1084758420101120 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067467016770560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3915080500294405 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3900961684844704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14118840877669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36110159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7848638525633874e+09 _step‚1130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3916644228903936 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77423270 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 _runtime‚24615.920990678 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17291035239424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3916642857287168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 243528033 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117678367 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3916644228903936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41313468 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825êšU¥4½8/ _wandbruntime‚24615  _runtime‚24615 eo¦¾8‹ %loss/total_loss‚7.683619914550731 epoch‚48 trainer/global_step‚13549 _runtime‚24615.955915423 % _timestamp‚1.7848638525986912e+09 _step‚1131 0loss/FRt2/assignment_loss‚0.7005142895173007 3loss/SRqqt1/assignment_loss‚0.49604633503090373 2loss/SRqqt2/assignment_loss‚0.4966453646295779 4loss/FBt1/assignment_loss‚0.00022103874659773481 2loss/FBt2/assignment_loss‚0.023857225402676964 /loss/FRt1/detection_loss‚0.4693894324856824 1loss/SRqqt2/detection_loss‚0.5816074626160005 0loss/FBt2/detection_loss‚0.37072374443773537 0loss/FRt1/assignment_loss‚1.5788176484968992 /loss/FRt2/detection_loss‚1.4109763206400845 1loss/SRqqt1/detection_loss‚1.0135262879305134 /loss/FBt1/detection_loss‚0.5412947646167595j¢¿8 3loss/SRqqt1/assignment_loss‚0.49604633503090373 2loss/SRqqt2/assignment_loss‚0.4966453646295779 4loss/FBt1/assignment_loss‚0.00022103874659773481 2loss/FBt2/assignment_loss‚0.023857225402676964 /loss/FRt1/detection_loss‚0.4693894324856824 1loss/SRqqt2/detection_loss‚0.5816074626160005 0loss/FBt2/detection_loss‚0.37072374443773537 0loss/FRt1/assignment_loss‚1.5788176484968992 /loss/FRt2/detection_loss‚1.4109763206400845 1loss/SRqqt1/detection_loss‚1.0135262879305134 /loss/FBt1/detection_loss‚0.5412947646167595 %loss/total_loss‚7.683619914550731 epoch‚48 trainer/global_step‚13549 _runtime‚24615.955915423 % _timestamp‚1.7848638525986912e+09 _step‚1131 0loss/FRt2/assignment_loss‚0.7005142895173007ë<‡áH’À8‚(:‡ ú¸‹Ó¬‹Î³ gpu.0.gpu‚69.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚110.067' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.689 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðíÂ2Á8‚(:§  ú¸‹Ó¬âë³ network.recv‚ 1.74173e+06 disk.drbd1965.in‚0 disk.drbd1584.in‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.046875* proc.memory.percent‚11.890554428100586 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2111.in‚0# disk.nvme0n1.out‚57099.19921875 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1965.out‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.sda1.out‚0) disk./.usagePercent‚70.48089331790382 disk.drbd1552.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚ 122.591796875 disk.drbd1931.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.sda.in‚0.046875( proc.memory.availableMB‚ 1908.2421875% disk./.usageGB‚1260.2699546813965 disk.dm-0.in‚ 143.78125 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1817.in‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1226.in‚0 disk.sda.out‚945 disk.drbd1289.in‚0 cpu‚25.342249431875043 disk.drbd1938.in‚0 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1584.out‚0 disk.sda2.out‚945# proc.memory.rssMB‚ 2922.22265625 disk.dm-0.out‚57099.19921875 disk.drbd2118.in‚0 network.sent‚ 2.3766063e+07 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.sda1.in‚0 disk.drbd1673.out‚0# memory_percent‚92.2353426615397 disk.drbd1552.out‚0Wf—‹Â8‚(:’ ‰¹‹Ó´ÊÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚293.807' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.93566666666668 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¿ÆIÃ8‚(:¾  ‰¹‹Óì“ä³ disk.drbd1817.out‚0 disk.drbd1931.in‚0& proc.memory.availableMB‚ 1908.21875 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1285.out‚0# disk.dm-0.out‚57099.49951171875 disk.drbd2078.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 945.44140625 network.sent‚ 2.376686e+07 network.recv‚ 1.741862e+06 disk.drbd1552.in‚0 disk.sda.in‚0.046875 disk.drbd1673.in‚0 disk.drbd1673.out‚0 disk.drbd2118.in‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 disk.drbd1550.out‚0* proc.memory.percent‚11.890554428100586& disk.nvme0n1.out‚57099.49951171875 disk.drbd1510.out‚0 disk.drbd2118.out‚0 disk.sda2.out‚ 945.44140625$ memory_percent‚92.23543802897134 disk.drbd2078.in‚0 disk.drbd2111.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1289.out‚0% disk./.usageGB‚1260.2699546813965 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.046875 disk.drbd1931.out‚0 cpu‚25.342203607785184 disk.drbd1356.in‚0 disk.drbd1510.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.48089331790382 disk.drbd1965.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚ 122.591796875 disk.drbd1550.in‚0 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 143.78125 disk.drbd1356.out‚0 disk.drbd1584.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2922.22265625àþ›JÄ8‚(:¿  ˜¹‹ÓàæÞ³ disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.drbd1356.in‚0 disk.drbd1510.out‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd2078.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1828.out‚0 disk.drbd1285.out‚0& disk.nvme0n1.out‚57099.91357421875 disk.sda.in‚0.046875 network.recv‚ 1.743671e+06 disk.drbd2041.out‚0 disk.rbd1.out‚122.6240234375 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1938.out‚0* proc.memory.percent‚11.890554428100586 disk.drbd1584.in‚0# disk.dm-0.out‚57099.91357421875 disk.drbd1226.in‚0 disk.sda2.in‚0.046875 disk.drbd2078.out‚0 network.sent‚ 2.377209e+07 disk.drbd1828.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 cpu‚25.342167936481616 disk.drbd2111.out‚0 disk.drbd1550.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1673.in‚0 disk.drbd1965.in‚0) disk./.usagePercent‚70.48089865134986 disk.drbd1673.out‚0 disk.sda2.out‚ 945.6640625 disk.drbd1931.in‚0 disk.drbd1289.in‚0( proc.memory.availableMB‚ 1907.9765625# proc.memory.rssMB‚ 2922.22265625% disk./.usageGB‚1260.2700500488281 disk.drbd1356.out‚0 disk.drbd1285.in‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0 disk.sda.out‚ 945.6640625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 143.78125 disk.drbd1938.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 143.78125 disk.drbd2111.in‚0 disk.drbd1552.in‚0 disk.drbd2118.out‚0$ memory_percent‚92.23642349243164—_Å{žÅ8‚(:“ ˜¹‹Óë¼³ gpu.0.gpu‚97.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚63.0 gpu.0.powerWatts‚130.206' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚43.401999999999994 gpu.0.smClock‚1380 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»NhFžÆ8‚(:“ §¹‹Ó³è³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.687' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.22900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßS‘ALÇ8‚(:Á  §¹‹Óƒ´ disk.drbd1673.out‚0 disk.drbd2041.in‚0 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0* proc.memory.percent‚11.890554428100586 disk.drbd1552.out‚0 disk.sda1.out‚0 disk.drbd1584.in‚0 disk.drbd1931.in‚0 disk.drbd2078.in‚0 disk.rbd1.out‚122.8115234375 disk.drbd1510.out‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0) proc.memory.availableMB‚ 1908.22265625) disk./.usagePercent‚70.48089865134986 disk.drbd1289.in‚0 disk.drbd1938.in‚0 disk.rbd1.in‚0 disk.drbd1226.out‚0& disk.nvme0n1.out‚57100.25927734375# disk.dm-0.out‚57100.25927734375 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0 disk.drbd2041.out‚0 cpu‚25.3421113806219 network.recv‚ 1.743845e+06 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 946.05078125$ memory_percent‚92.23542213439941# proc.memory.rssMB‚ 2922.22265625 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 143.78125 disk.dm-0.in‚ 143.78125 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1510.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 946.05078125 disk.drbd1285.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.046875 network.sent‚ 2.3774419e+07 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.046875 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.drbd1828.out‚0% disk./.usageGB‚1260.2700500488281õW‰È8‚(:’ ¶¹‹Óħª³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.096' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.03200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùª"›NÉ8‚(:à  ¶¹‹Ó¥ôij disk.drbd1673.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd2118.in‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.046875 disk.drbd1356.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1550.in‚0 disk.sda2.out‚ 946.30078125 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 cpu‚25.342076760491782 disk.drbd1584.in‚0 disk.drbd2041.in‚0* proc.memory.percent‚11.890554428100586 disk.drbd1817.out‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚57100.94287109375$ memory_percent‚92.23542213439941 disk.drbd1510.out‚0 disk.drbd1965.in‚0) proc.memory.availableMB‚ 1908.22265625 disk.drbd1226.in‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.sda.out‚ 946.30078125 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.dm-0.in‚ 143.78125 disk.drbd1938.out‚0& disk.nvme0n1.out‚57100.94287109375 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.046875 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1552.out‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0 disk.drbd2078.in‚0 disk.drbd1931.in‚0 disk.drbd2041.out‚0 disk.drbd2111.in‚0 network.sent‚ 2.3778301e+07 network.recv‚ 1.745841e+06% disk./.usageGB‚1260.2699737548828 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.rbd1.out‚122.8740234375) disk./.usagePercent‚70.48089438459304 disk.sda1.in‚0‰my¾4Ê8/ _wandbruntime‚24700  _runtime‚24700/iõ ÊË8Ä _runtime‚24700.969552889 _step‚1132 & lr-AdamW/pg1‚2.467059632285201e-07 & lr-AdamW/pg2‚2.467059632285201e-07 trainer/global_step‚13599 % _timestamp‚1.7848639376123266e+09Ö^¦ƒÏÌ8É _step‚1132 & lr-AdamW/pg1‚2.467059632285201e-07 & lr-AdamW/pg2‚2.467059632285201e-07 trainer/global_step‚13599 % _timestamp‚1.7848639376123266e+09 _runtime‚24700.969552889ì³½R¥4Í8/ _wandbruntime‚24700  _runtime‚24700·RµV¤Î8˜P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17352005991424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77696788 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36239524 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14170268570123 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126286458 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚13599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14192688072192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3928791870290597 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118088350 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118088350 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3916166891497472 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3916166891497472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118088344 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1088669754693632 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3914622946331760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126296522 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/requ†&Måz ested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14192688072192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3930360951179264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1071318116793344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3914622971759728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244374802 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3916166917670912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126286458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚372 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1071317748702208 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3930359579569664ûßz#(Ï8P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3916166891497472 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3916166891497472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118088344 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1088669754693632 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3914622946331760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126296522 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14192688072192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3930360951179264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 429843968 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1071318116793344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3914622971759728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244374802 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3916166917670912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126286458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚372 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1071317748702208 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3930359579569664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17352005991424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77696788 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36239524 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14170268570123 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126286458 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚13599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61752832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14192688072192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3928791870290597 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118088350 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118088350 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1í™n 4Ð8/ _wandbruntime‚24702  _runtime‚24702Ç;/''Ñ8¡N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244402335 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17353300559360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3914901748105536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41460604 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17353243008000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126295540 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126295540 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1071396703159296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 % _timestamp‚1.7848639390918276e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1134 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3914901773533504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126305618 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36242159 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3929073062905379 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118096711 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3930639511368192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14171289371875 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚13599 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118096717 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14193710580224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14193710580224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3929071692866013 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244392251 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126305618 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244402335 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3916445800787968 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3916445800787968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1088749578076160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77702393 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36242164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚24702.44943233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3916445826961408 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77702768 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3930640882984960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118096711 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1088750003718656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244392251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14195056023552Êå&Ò8¦N PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1088749578076160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77702393 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36242164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 _runtime‚24702.44943233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3916445826961408 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77702768 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3930640882984960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118096711 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1088750003718656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244392251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14195056023552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244402335 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17353300559360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3914901748105536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41460604 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17353243008000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57551360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126295540 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126295540 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1071396703159296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 % _timestamp‚1.7848639390918276e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1134 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3914901773533504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126305618 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36242159 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3929073062905379 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118096711 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3930639511368192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14171289371875 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚13599 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118096717 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14193710580224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14193710580224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3929071692866013 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244392251 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126305618 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244402335 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3916445800787968 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3916445800787968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 425642496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_¨DGalloc‚2006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0îm|;Œ4Ó8/ _wandbruntime‚24702  _runtime‚24702‡»KÔ8ˆ 2loss/FBt1/assignment_loss‚0.004869633551638187 /loss/FRt2/detection_loss‚1.3740850507894604 1loss/SRqqt2/detection_loss‚0.5692749594905188 epoch‚48 % _timestamp‚1.7848639391270797e+09 _runtime‚24702.484346563 2loss/SRqqt2/assignment_loss‚0.4731459860711221 1loss/SRqqt1/detection_loss‚0.8043429729355595 /loss/FBt1/detection_loss‚0.6374628992581909 %loss/total_loss‚7.469538580505854 _step‚1135 0loss/FRt1/assignment_loss‚1.5516322798775015 /loss/FRt1/detection_loss‚0.5299201462409985 0loss/FRt2/assignment_loss‚0.6464062425235954 2loss/SRqqt1/assignment_loss‚0.5087186389673195 2loss/FBt2/assignment_loss‚0.010017076825730211 0loss/FBt2/detection_loss‚0.35966269397421885 trainer/global_step‚13599«Yç?Õ8 _runtime‚24702.484346563 2loss/SRqqt2/assignment_loss‚0.4731459860711221 1loss/SRqqt1/detection_loss‚0.8043429729355595 /loss/FBt1/detection_loss‚0.6374628992581909 %loss/total_loss‚7.469538580505854 _step‚1135 0loss/FRt1/assignment_loss‚1.5516322798775015 /loss/FRt1/detection_loss‚0.5299201462409985 0loss/FRt2/assignment_loss‚0.6464062425235954 2loss/SRqqt1/assignment_loss‚0.5087186389673195 2loss/FBt2/assignment_loss‚0.010017076825730211 0loss/FBt2/detection_loss‚0.35966269397421885 trainer/global_step‚13599 2loss/FBt1/assignment_loss‚0.004869633551638187 /loss/FRt2/detection_loss‚1.3740850507894604 1loss/SRqqt2/detection_loss‚0.5692749594905188 epoch‚48 % _timestamp‚1.7848639391270797e+09ïûU\ùÖ8‚(:’ Ź‹ÓÐÑг gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.923' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.97433333333333 gpu.0.smClock‚1335 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šQÙF×8‚(:»  Ź‹ÓËÂñ³ disk.drbd1938.in‚0 disk.drbd1584.out‚0 disk.drbd1673.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 143.78125 disk.drbd1828.out‚0 disk.drbd1226.in‚0 disk.drbd1356.out‚0" disk.nvme0n1.out‚ 57101.4765625 disk.sda.in‚0.046875 network.recv‚ 1.746039e+06 disk.drbd1552.in‚0 disk.drbd1285.out‚0 disk.drbd1965.in‚0 disk.drbd2041.out‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.046875 disk.drbd2078.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 946.66796875 cpu‚25.342030284211095 disk.drbd2041.in‚0 disk.dm-0.out‚ 57101.4765625 disk.sda1.in‚0* proc.memory.percent‚11.890554428100586 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd1510.in‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚122.8740234375 disk.drbd1226.out‚0 disk.drbd1931.out‚0 disk.drbd1289.out‚0 disk.drbd2111.in‚0 disk.drbd2118.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 946.66796875$ memory_percent‚92.23570823669434 disk.drbd1931.in‚0 disk.drbd1965.out‚0 disk.drbd2111.out‚0 disk.drbd1673.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.drbd1584.in‚0 network.sent‚ 2.3781129e+07) proc.memory.availableMB‚ 1908.15234375 disk.drbd1510.out‚0 disk.drbd1552.out‚0) disk./.usagePercent‚70.48089438459304% disk./.usageGB‚1260.2699737548828P¿žØ8‚(:“ Ô¹‹Ó阻³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚313.891' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚104.63033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úØ‹=Ù8‚(:²  Ô¹‹ÓäåÚ³) disk./.usagePercent‚70.48089438459304 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd2118.in‚0 network.recv‚ 1.746147e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 122.90625 disk.drbd1828.out‚0 disk.drbd1550.out‚0 disk.drbd1356.out‚0 disk.drbd2041.in‚0# memory_percent‚92.2356923421224 disk.drbd2078.out‚0 disk.drbd2111.in‚0 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 cpu‚25.341964797399292 proc.cpu.threads‚17 disk.drbd1931.out‚0* proc.memory.percent‚11.890554428100586 disk.drbd1510.in‚0 disk.drbd2118.out‚0 disk.sda.out‚ 946.9296875 disk.drbd1673.in‚0 disk.sda2.out‚ 946.9296875 disk.dm-0.out‚57101.70703125 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1817.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1938.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.046875 disk.dm-0.in‚ 143.78125 disk.drbd1550.in‚0 disk.sda2.in‚0.046875 disk.drbd1289.in‚0% disk./.usageGB‚1260.2699737548828# disk.nvme0n1.out‚57101.70703125 disk.drbd1552.out‚0 disk.drbd1817.out‚0 network.sent‚ 2.3781237e+07 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1552.in‚0 disk.drbd1931.in‚0& proc.memory.availableMB‚ 1908.15625ˆ'®áDÚ8‚(:¹  㹋Ó­ùý³) disk./.usagePercent‚70.48073992799627 disk.drbd1931.out‚0 disk.drbd1938.in‚0# disk.dm-0.out‚57102.33935546875 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.sda.out‚ 947.3046875 disk.drbd1673.in‚0 disk.drbd2078.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚57102.33935546875 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.046875 disk.drbd1356.in‚0 disk.drbd1965.in‚0 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1552.in‚0 disk.drbd2078.out‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 143.78125 cpu‚25.34192870433629 network.sent‚ 2.3844385e+07 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1817.out‚0 disk.drbd1938.out‚0% disk./.usageGB‚1260.2672119140625 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚92.2356923421224 disk.dm-0.in‚ 143.78125 disk.drbd2041.in‚0 disk.rbd1.out‚ 123.0078125 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.sda2.out‚ 947.3046875 disk.drbd1289.out‚0& proc.memory.availableMB‚ 1908.15625 disk.drbd1285.in‚0 disk.drbd2118.in‚0 disk.sda.in‚0.046875 disk.drbd1931.in‚0 network.recv‚ 1.749055e+06 disk.drbd1356.out‚0* proc.memory.percent‚11.890554428100586 disk.drbd1828.in‚0 disk.drbd1965.out‚0 proc.cpu.threads‚174À)äžÛ8‚(:“ 㹋Óø’Ú³ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚307.625' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.54166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¼*¹Ü8‚(:’ ò¹‹Óéßȳ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚294.997' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.33233333333334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C}¦Y8Ý8‚(:­  ò¹‹Ó¶·é³ disk.drbd1289.in‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0$ memory_percent‚92.23664601643881 disk.sda2.out‚947.625 disk.drbd2078.out‚0 disk.drbd1510.out‚0 disk.drbd1226.in‚0 disk.sda.out‚947.625 disk.drbd1673.in‚0$ disk./.usageGB‚1260.267162322998 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd2118.out‚0 disk.dm-0.in‚ 143.78125 disk.drbd1356.out‚0 disk.drbd1285.out‚0# disk.nvme0n1.out‚57102.51953125 cpu‚25.341913754606495 network.sent‚ 2.3844451e+07 disk.rbd1.out‚ 123.0703125 disk.nvme0n1.in‚ 143.78125 disk.drbd2078.in‚0 disk.drbd2041.out‚0 disk.drbd1828.out‚0 network.recv‚ 1.749121e+06 disk.drbd1552.in‚0 disk.sda.in‚0.046875 disk.drbd1931.in‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0' proc.memory.availableMB‚ 1907.921875* proc.memory.percent‚11.890554428100586) disk./.usagePercent‚70.48073715460433 disk.sda1.in‚0 disk.drbd1828.in‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.sda2.in‚0.046875 disk.drbd1931.out‚0 disk.dm-0.out‚57102.51953125 disk.sda1.out‚0Ö~ð”“Þ8‚(:ˆ º‹Ó‡’¾³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.732' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚102.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹yÙEß8‚(:º  º‹Óí¨Ý³ disk.drbd1584.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.sda1.out‚0 disk.drbd1510.out‚0% disk./.usageGB‚1260.2671813964844 disk.sda2.out‚ 948.015625 disk.drbd1226.out‚0" disk.dm-0.out‚57103.0849609375 disk.drbd2118.out‚0) proc.memory.availableMB‚ 1908.51171875* proc.memory.percent‚11.890554428100586 disk.drbd1356.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.046875 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.sda.out‚ 948.015625 disk.drbd2078.in‚0% disk.nvme0n1.out‚57103.0849609375 proc.cpu.threads‚17 disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.drbd1285.out‚0 disk.drbd1552.out‚0 disk.sda1.in‚0$ memory_percent‚92.23424593607584 disk.drbd1356.out‚0) disk./.usagePercent‚70.48073822129354 disk.drbd1673.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 network.recv‚ 1.750178e+06 disk.drbd1938.out‚0 cpu‚25.341858360583736 disk.sda2.in‚0.046875 disk.drbd1828.in‚0 disk.drbd1226.in‚0 network.sent‚ 2.3847007e+07 disk.drbd1673.in‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.drbd2041.in‚0 disk.dm-0.in‚ 143.78125 disk.drbd1965.out‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 123.0703125 disk.nvme0n1.in‚ 143.78125 disk.rbd1.in‚0 disk.drbd2118.in‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0óAñà8‚(:’ º‹Óíöº³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚281.398' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.79933333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09-çGá8‚(:¼  º‹ÓœÞ³ disk.drbd1584.in‚0 disk.sda2.out‚ 948.296875 disk.drbd2041.in‚0% disk.nvme0n1.out‚57103.2607421875 disk.sda1.out‚0" disk.dm-0.out‚57103.2607421875 disk.drbd1285.in‚0 disk.drbd1931.out‚0 disk.drbd1938.out‚0) proc.memory.availableMB‚ 1908.75390625) disk./.usagePercent‚70.48073715460433 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.23326047261557 network.sent‚ 2.3847115e+07 disk.drbd2111.in‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1510.out‚0 disk.drbd1550.in‚0 disk.drbd1931.in‚0 disk.drbd1510.in‚0$ disk./.usageGB‚1260.267162322998 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.046875 disk.sda.out‚ 948.296875 disk.sda2.in‚0.046875 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd2118.out‚0 disk.drbd1673.out‚0 cpu‚25.341822967962305 disk.drbd1584.out‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.rbd1.in‚0 disk.drbd1289.out‚0 disk.rbd1.out‚123.1025390625 disk.drbd1226.in‚0 disk.drbd1828.out‚0 disk.dm-0.in‚ 143.78125 disk.drbd2078.in‚0 disk.drbd1673.in‚0 disk.drbd2041.out‚0 disk.drbd1285.out‚0 proc.cpu.threads‚17 network.recv‚ 1.750286e+06 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2922.22265625* proc.memory.percent‚11.890554428100586 disk.drbd1552.out‚0 disk.sda1.in‚0QRÊô4â8/ _wandbruntime‚24787  _runtime‚24787&×ãÊã8Ä trainer/global_step‚13649 % _timestamp‚1.7848640244285614e+09 _runtime‚24787.786889965 _step‚1136 & lr-AdamW/pg1‚1.890499173242642e-07 & lr-AdamW/pg2‚1.890499173242642e-07(»Æ§Ïä8É _step‚1136 & lr-AdamW/pg1‚1.890499173242642e-07 & lr-AdamW/pg2‚1.890499173242642e-07 trainer/global_step‚13649 % _timestamp‚1.7848640244285614e+09 _runtime‚24787.786889965ðk‚áï4å8/ _wandbruntime‚24787  _runtime‚24787?ŒÏ­(æ8—P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126752474 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1075247435090944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3944356233650688 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118506694 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3930112382195712 trainer/global_step‚13649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3928563035020560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118506700 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚24787.786996456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3944356233650688 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3930112382195712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77976815 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3930112356022272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36371519 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚1137 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126752474 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3942783062862205 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14243877628416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1075247066999808 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3942784432901459 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1092661023735808 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3930112356022272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36371524 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1092661449385472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14220027841645 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14245223064576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14221372452931 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245249104 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3944357605260288 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14245223064576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17413956736000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 $ _timestamp‚1.784864024429339e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126742410 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41605296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664’©{ç8œP N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 245259174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17414014294528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118506694 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126742410 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3930112356022272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36371519 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚1137 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126752474 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3942783062862205 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14243877628416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1075247066999808 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3942784432901459 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1092661023735808 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3930112356022272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36371524 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1092661449385472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14220027841645 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14245223064576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14221372452931 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245249104 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3944357605260288 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14245223064576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17413956736000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 $ _timestamp‚1.784864024429339e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126742410 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41605296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 425649664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126752474 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1075247435090944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3944356233650688 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118506694 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3930112382195712 trainer/global_step‚13649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3928563035020560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118506700 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 _runtime‚24787.786996456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.fFnyP§reed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3944356233650688 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3930112382195712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77976815 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚391 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17415196779520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14246245579776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77982391 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36374159 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1092740850145280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17415262719488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245276637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126761570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3930391265312768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118515061 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3930391291486208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245266553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118515061 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1092741284176384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3928841862222304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 _step‚1138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3944637537065984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245276637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126761570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3930391265312768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118515061 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3930391291486208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245266553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118515061 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1092741284176384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3928841862222304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 _step‚1138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3944637537065984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚391 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17415196779520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14246245579776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77982391ò¨Áí4ë8/ _wandbruntime‚24789  _runtime‚24789u÷ ì8† 3loss/SRqqt2/assignment_loss‚0.43186479168179487 1loss/SRqqt2/detection_loss‚0.5692330145632094 trainer/global_step‚13649 $ _timestamp‚1.784864025940234e+09 _runtime‚24789.297477154 _step‚1139 3loss/FBt1/assignment_loss‚3.185402525271996e-05 2loss/FBt2/assignment_loss‚0.003539615609745585 /loss/FBt2/detection_loss‚0.5537575218246968 epoch‚48 /loss/FRt1/assignment_loss‚1.619358416273458 0loss/FRt2/assignment_loss‚0.6480400147702959 2loss/SRqqt1/assignment_loss‚0.5970288213769732 /loss/FRt1/detection_loss‚0.5090767771379413 .loss/FRt2/detection_loss‚1.357565032023833 1loss/SRqqt1/detection_loss‚0.8401612078246735 /loss/FBt1/detection_loss‚0.5657846249552676 %loss/total_loss‚7.695441692067142Åósòí8‹ 0loss/FRt2/assignment_loss‚0.6480400147702959 2loss/SRqqt1/assignment_loss‚0.5970288213769732 /loss/FRt1/detection_loss‚0.5090767771379413 .loss/FRt2/detection_loss‚1.357565032023833 1loss/SRqqt1/detection_loss‚0.8401612078246735 /loss/FBt1/detection_loss‚0.5657846249552676 %loss/total_loss‚7.695441692067142 3loss/SRqqt2/assignment_loss‚0.43186479168179487 1loss/SRqqt2/detection_loss‚0.5692330145632094 trainer/global_step‚13649 $ _timestamp‚1.784864025940234e+09 _runtime‚24789.297477154 _step‚1139 3loss/FBt1/assignment_loss‚3.185402525271996e-05 2loss/FBt2/assignment_loss‚0.003539615609745585 /loss/FBt2/detection_loss‚0.5537575218246968 epoch‚48 /loss/FRt1/assignment_loss‚1.619358416273458ó¯üòËœî8‚(:‘ Ÿº‹Óµ’Ò³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚306.057' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.01900000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«|Õ?ï8‚(:´  Ÿº‹ÓÊ¥ñ³# proc.memory.rssMB‚ 2922.22265625 disk.drbd1938.out‚0 disk.drbd2078.out‚0 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.sda2.out‚ 948.6796875 disk.drbd1931.out‚0 disk.dm-0.out‚ 57103.671875 disk.drbd1931.in‚0 disk.drbd1226.out‚0) disk./.usagePercent‚70.48076275514524 disk.drbd1828.out‚0 disk.rbd1.out‚123.2041015625 disk.drbd1552.out‚0 disk.drbd2118.in‚0 proc.cpu.threads‚17! disk.nvme0n1.out‚ 57103.671875 disk.drbd2041.out‚0 disk.drbd2118.out‚0$ memory_percent‚92.23338762919107 disk.drbd1965.in‚0 disk.rbd1.in‚0 disk.drbd1584.out‚0 disk.sda.in‚0.046875 network.recv‚ 1.750352e+06 disk.drbd1938.in‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1908.72265625 network.sent‚ 2.3847181e+07 disk.drbd2078.in‚0 disk.drbd1356.in‚0 disk.drbd1828.in‚0 disk.sda.out‚ 948.6796875 cpu‚25.34178731636508 disk.drbd1356.out‚0 disk.sda1.out‚0 disk.drbd1673.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0* proc.memory.percent‚11.890554428100586# disk./.usageGB‚1260.26762008667 disk.dm-0.in‚ 143.78125 disk.drbd2111.out‚0 disk.sda2.in‚0.046875 disk.drbd1673.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd2041.in‚0 disk.nvme0n1.in‚ 143.78125 disk.drbd1510.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0aò„.’ð8‚(:‡ ®º‹Óÿè´³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚288.159' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I#üÅ8ñ8‚(:­  ®º‹Ó’׳ disk.drbd1552.in‚0 proc.cpu.threads‚17 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.dm-0.out‚ 57103.8125 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 143.78125) proc.memory.availableMB‚ 1908.72265625# proc.memory.rssMB‚ 2922.22265625 disk.drbd2078.out‚0 disk.drbd1584.in‚0 disk.drbd1817.out‚0* proc.memory.percent‚11.890554428100586 disk.nvme0n1.out‚ 57103.8125$ memory_percent‚92.23338762919107 disk.drbd1356.in‚0 disk.rbd1.out‚123.2666015625 disk.drbd1510.in‚0 network.sent‚ 2.3850745e+07 disk.drbd2078.in‚0 disk.sda2.in‚0.046875 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.drbd1289.out‚0 disk.drbd2111.out‚0 disk.drbd1226.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1931.out‚0 disk.drbd1828.out‚0 disk.drbd1584.out‚0 disk.sda.out‚ 948.921875 disk.drbd1673.out‚0 cpu‚25.3417222691301 disk.drbd1931.in‚0 disk.drbd1285.out‚0) disk./.usagePercent‚70.48076275514524 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.drbd1550.in‚0 disk.sda1.in‚0 network.recv‚ 1.751473e+06 disk.sda2.out‚ 948.921875 disk.dm-0.in‚ 143.78125 disk.drbd1285.in‚0 disk.sda.in‚0.046875# disk./.usageGB‚1260.26762008667Cùj>ò8‚(:’ ½º‹ÓñØ˳ gpu.0.gpu‚89.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚261.769' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.25633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Së…GLó8‚(:Á  ½º‹Ó‡é³ disk.drbd1356.in‚0) disk./.usagePercent‚70.48098014640506 disk.sda2.out‚ 949.46875# disk.dm-0.out‚57104.14404296875 disk.drbd1965.in‚0 disk.drbd2118.in‚0) proc.memory.availableMB‚ 1908.72265625# proc.memory.rssMB‚ 2922.22265625 disk.drbd2111.out‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 cpu‚25.341706846934162 disk.sda.out‚ 949.46875 disk.dm-0.in‚ 144.8828125* proc.memory.percent‚11.890554428100586 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚123.2666015625 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda.in‚0.046875 disk.sda2.in‚0.046875 disk.drbd1938.in‚0 disk.drbd2041.out‚0 disk.drbd1552.out‚0$ memory_percent‚92.23338762919107 network.sent‚ 2.3850853e+07 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1285.in‚0 disk.nvme0n1.in‚ 144.8828125& disk.nvme0n1.out‚57104.14404296875 disk.drbd1931.in‚0 disk.drbd1289.in‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd1510.out‚0 disk.drbd1584.out‚0 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1550.out‚0 network.recv‚ 1.751581e+06 disk.drbd2111.in‚0 disk.drbd1817.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0% disk./.usageGB‚1260.2715072631836 disk.drbd1673.out‚0 disk.sda1.out‚0 disk.drbd1828.in‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17õ]òœô8‚(:‘ ̺‹ÓÞËij gpu.0.gpu‚38.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.73' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.91000000000001 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j>p§Oõ8‚(:Ä  ̺‹Ó„³å³ disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0 disk.rbd1.out‚ 123.298828125 disk.nvme0n1.in‚ 144.8828125# proc.memory.rssMB‚ 2922.22265625 disk.drbd1938.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1356.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.046875" disk.dm-0.out‚57104.6962890625% disk.nvme0n1.out‚57104.6962890625 disk.drbd1226.in‚0) proc.memory.availableMB‚ 1908.72265625 disk.drbd2078.in‚0 disk.drbd1510.out‚0 network.sent‚ 2.3850961e+07 disk.drbd1817.in‚0 disk.drbd1931.in‚0 disk.drbd1550.out‚0 cpu‚25.341671709989853 disk.sda.out‚ 949.71484375 disk.drbd1289.out‚0 disk.drbd2041.in‚0 disk.sda.in‚0.046875$ memory_percent‚92.23338762919107 network.recv‚ 1.751689e+06 disk.drbd1226.out‚0 proc.cpu.threads‚17 disk.drbd1673.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.2714729309082 disk.sda2.out‚ 949.71484375 disk.drbd1931.out‚0 disk.drbd1828.out‚0* proc.memory.percent‚11.890554428100586 disk.sda1.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.48097822636448 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0ÁBë–ö8‚(:½  Ûº‹Ó³—Ò³ network.sent‚ 2.3924541e+07 disk.drbd1356.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1226.in‚0 disk.drbd1552.out‚0 disk.sda.in‚0.046875 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1289.out‚0 disk.drbd1938.in‚0 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.sda2.out‚ 950.1953125& proc.memory.availableMB‚ 1908.71875 proc.cpu.threads‚17" disk.dm-0.out‚57105.3896484375 disk.drbd2041.out‚0 disk.drbd1285.out‚0 disk.drbd1510.out‚0 disk.drbd1817.out‚0 disk.drbd2041.in‚0 network.recihª²v‚ 1.755341e+06 disk.drbd1965.out‚0% disk./.usageGB‚1260.2715492248535 disk.drbd1828.in‚0 disk.drbd1552.in‚0 cpu‚25.34160674967768 disk.drbd1931.out‚0 disk.drbd1938.out‚0 disk.drbd2118.out‚0 disk.sda.out‚ 950.1953125 disk.drbd1289.in‚0 disk.drbd1584.in‚0 disk.drbd1584.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0$ memory_percent‚92.23340352376303 disk.drbd1931.in‚0 disk.dm-0.in‚ 144.8828125 disk.rbd1.out‚ 123.580078125 disk.drbd1550.in‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.890554428100586 disk.drbd1356.out‚0 disk.drbd1226.out‚0 disk.drbd2111.out‚0# proc.memory.rssMB‚ 2922.22265625 disk.drbd1965.in‚0 disk.drbd2078.out‚0% disk.nvme0n1.out‚57105.3896484375( disk./.usagePercent‚70.4809824931213 disk.sda2.in‚0.046875 disk.nvme0n1.in‚ 144.8828125 disk.drbd1510.in‚0 disk.drbd1550.out‚0Ô§2ž÷8‚(:“ Ûº‹Óε³ gpu.0.gpu‚98.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.245' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.08166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Ýožø8‚(:“ 꺋Óˆ¢Á³ gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚300.809' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚100.26966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ‘Hù8‚(:½  꺋Ó™÷ä³ disk.drbd2111.in‚0# proc.memory.rssMB‚ 2922.22265625% proc.memory.availableMB‚ 1908.6875 network.sent‚ 2.3926825e+07 disk.drbd2078.out‚0 disk.drbd1550.out‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.drbd2078.in‚0 disk.drbd1828.out‚0$ disk.nvme0n1.out‚57106.017578125 disk.drbd1931.out‚0 cpu‚25.34157132821038 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1938.out‚0! disk.dm-0.out‚57106.017578125 disk.drbd2041.out‚0 disk.drbd1550.in‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1817.out‚0 disk.drbd1938.in‚0$ memory_percent‚92.23353068033853 disk.nvme0n1.in‚ 144.8828125 disk.drbd1584.out‚0 disk.sda.out‚ 950.69140625 disk.drbd1289.in‚0% disk./.usageGB‚1260.2687683105469 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1552.out‚0 disk.drbd1584.in‚0 disk.sda2.in‚0.046875 disk.drbd1965.out‚0 network.recv‚ 1.755473e+06) disk./.usagePercent‚70.48082696983533 disk.drbd2041.in‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.890554428100586 disk.rbd1.out‚ 123.642578125 disk.drbd1226.in‚0 disk.drbd1817.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1965.in‚0 disk.drbd1356.out‚0 disk.drbd2111.out‚0 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.sda.in‚0.046875 disk.drbd1673.in‚0 disk.sda2.out‚ 950.69140625›5Â4ú8/ _wandbruntime‚24874  _runtime‚248744çOZÊû8Ä trainer/global_step‚13699 $ _timestamp‚1.784864111373872e+09 _runtime‚24874.73115409 _step‚1140 ' lr-AdamW/pg1‚1.3904555698795487e-07 ' lr-AdamW/pg2‚1.3904555698795487e-07ø‚gÏü8É $ _timestamp‚1.784864111373872e+09 _runtime‚24874.73115409 _step‚1140 ' lr-AdamW/pg1‚1.3904555698795487e-07 ' lr-AdamW/pg2‚1.3904555698795487e-07 trainer/global_step‚13699ô.KÙ4ý8/ _wandbruntime‚24874  _runtime‚24874i€(þ8˜P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127208426 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3944057846720512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 419358208 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1096652601318912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17476216021504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118925044 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78257045 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118925050 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14296412620800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3958352887731712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚375 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1096653020677120 _runtime‚24874.735928136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3958354259341312 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3944057846720512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78256665 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14272476335739 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 trainer/global_step‚13699 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14296412620800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246123406 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36503519 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41753521 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3958352887731712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 _step‚1141 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1079176753388544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36503524 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1079176385297408 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3956774255433813 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3944057820547072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3958354259341312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118925050 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3956775625473067 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3942503149137328 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3944057820547072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17476267288576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 % _timestamp‚1.7848641113750176e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78257045 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118925050 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14296412620800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3958352887731712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚375 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1096653020677120 _runtime‚24874.735928136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51267072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3958354259341312 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3944057846720512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78256665 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14272476335739 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 trainer/global_step‚13699 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14296412620800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246123406 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36503519 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41753521 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3958352887731712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 _step‚1141 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1079176753388544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36503524 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1079176385297408 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3956774255433813 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3944057820547072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3958354259341312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118925050 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3956775625473067 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3942503149137328 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3944057820547072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17476267288576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 % _timestamp‚1.7848641113750176e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3944336756011008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3944336756011008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41756068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17477468825088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14272152526093 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1079254971663360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17477520084992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13699 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3958634191147008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127207444 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1079255339754496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3942781925483136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _step‚1142 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14296089692672 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3958634191147008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78262227 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246140855 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118933417 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246150939 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36506164 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14297435136000 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1096732440488448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41756440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78262604 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006÷îjb-'‚9§N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246150939 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78262227 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246140855 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118933417 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246150939 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36506164 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14297435136000 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1096732440488448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41756440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78262604 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 _runtime‚24876.208618893 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14297435136000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14296089692672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118933411 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246140855 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1096732859839488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3944336729837568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3957055448048595 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 % _timestamp‚1.7848641128510106e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127217522 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127207444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3944336756011008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3944336756011008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41756068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17477468825088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14272152526093 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1079254971663360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17477520084992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13699 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3958634191147008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127207444 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1079255339754496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3942781925483136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 _step‚1142 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14296089692672 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3958634191147008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 FêMŒ9‚(:  ¦»‹Ó×’ø³ network.sent‚ 2.3991705e+07 disk.drbd1289.in‚0 disk.rbd1.out‚ 124.048828125) disk./.usagePercent‚70.48082654315965 disk.drbd1584.in‚0 disk.sda2.in‚0.046875 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd2078.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1552.in‚0 disk.drbd1828.out‚0& proc.memory.availableMB‚ 1907.65625 cpu‚25.34332379559777 disk.drbd2078.in‚0 disk.drbd1550.out‚0$ memory_percent‚92.23772684733072 disk.drbd1817.out‚0 network.recv‚ 1.759764e+06 disk.drbd1550.in‚0 proc.cpu.threads‚17 disk.drbd1584.out‚0 disk.sda.in‚0.046875 disk.sda.out‚ 952.14453125 disk.drbd1226.in‚0# disk.dm-0.out‚57108.20068359375 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 144.8828125 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2923.18359375 disk.drbd1673.out‚0 disk.drbd2041.in‚0% disk./.usageGB‚1260.2687606811523 disk.drbd1938.in‚0 disk.dm-0.in‚ 144.8828125 disk.drbd2041.out‚0& disk.nvme0n1.out‚57108.20068359375 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.drbd1965.out‚0 disk.sda2.out‚ 952.14453125 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.drbd1510.out‚0* proc.memory.percent‚11.894464492797852 disk.drbd1931.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0oƒG 9‚(:’ ¦»‹ÓãŽÙ³ gpu.0.gpu‚99.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚60.0 gpu.0.powerWatts‚106.802' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚35.60066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S94Ž9/ _wandbruntime‚24930  _runtime‚24930$7+9ˆv 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1796320676803589 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03302428871393204 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7456220984458923 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9640940427780151 )loss/val_total_loss‚7.717541601751094 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2972675859928131 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2699529230594635 *jet/accuracy_1_of_2‚0.7912917137145996 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7validation_average_jet_accuracy‚0.40683045983314514 trainer/global_step‚13719 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7942200303077698 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028090298175811768 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5164191722869873 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36926814913749695 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 particle/accuracy_1_of_1‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07369927316904068 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4590897262096405 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 particle/accuracy_2_of_2‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02649708092212677 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04384445399045944 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7160285711288452 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6916908621788025 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17894676327705383 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9832039475440979 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9130017161369324 _runtime‚24930.049623591 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3884442150592804 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.76884925365448 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.720253586769104 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14848235249519348 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2714218497276306 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6880441308021545 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9215782284736633 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6292827725410461 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6440386772155762 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7189925909042358 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.358066588640213 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7"Purity/0FRt*SRqqt1FBt/event_purity‚0.87553870677948 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6440386772155762 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2812763750553131 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5463929772377014 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3700268268585205 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2795344591140747 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6387069821357727 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3855293095111847 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.3006325364112854 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31206831336021423 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.881477952003479 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.632697582244873 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.358066588640213 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6402968168258667 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27049094438552856 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6548571586608887 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6630035638809204 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6585618257522583 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7848641666916096e+09 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 particle/accuracy_2_of_3‚1 'particle/accuracy‚0.854840874671936 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7763156890869141 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37166884541511536 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.794957160949707 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16337810456752777 )jet/accuracy_1_of_3‚0.938784658908844 epoch‚48 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17566870152950287 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35839056968688965 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22488434612751007 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08986023813486099 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22842039167881012 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6980623602867126 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5266430974006653 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6926719546318054 _step‚1144 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7517768740653992 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.781562864780426 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6773847937583923 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8085681200027466 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7211027145385742 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6911553144454956 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3493846356868744 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096c59I× ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053127773106098175 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35586875677108765 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6541314721107483 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5870780348777771 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40729379653930664 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3827584683895111 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8027581572532654 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *particle/sensitivity‚0.641911506652832 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6645923256874084 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38849857449531555 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6316066384315491 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36056089401245117 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7050897479057312 +jet/accuracy_2_of_2‚0.17263178527355194 *jet/accuracy_2_of_3‚0.6049410700798035 *jet/accuracy_2_of_4‚0.8373957276344299 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3095727860927582 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2897402346134186 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +jet/accuracy_3_of_3‚0.07885482162237167 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19163218140602112 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7172498106956482 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03082011267542839 *jet/accuracy_1_of_1‚0.3728956878185272\­ü;9v 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.76884925365448 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.720253586769104 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.14848235249519348 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2714218497276306 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6880441308021545 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9215782284736633 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6292827725410461 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6440386772155762 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7189925909042358 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.358066588640213 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7"Purity/0FRt*SRqqt1FBt/event_purity‚0.87553870677948 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6440386772155762 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2812763750553131 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5463929772377014 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3700268268585205 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2795344591140747 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6387069821357727 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3855293095111847 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.3006325364112854 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31206831336021423 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.881477952003479 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.632697582244873 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.358066588640213 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6402968168258667 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27049094438552856 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.018756305798888206 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6548571586608887 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6630035638809204 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6585618257522583 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7848641666916096e+09 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 particle/accuracy_2_of_3‚1 'particle/accuracy‚0.854840874671936 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7763156890869141 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9993678331375122 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.37166884541511536 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.794957160949707 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16337810456752777 )jet/accuracy_1_of_3‚0.938784658908844 epoch‚48 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17566870152950287 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.35839056968688965 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22488434612751007 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08986023813486099 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22842039167881012 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6980623602867126 +jet/accuracy_3_of_4‚0.10971427708864212 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5266430974006653 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6926719546318054 _step‚1144 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7517768740653992 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.781562864780426 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6773847937583923 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8085681200027466 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7211027145385742 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6911553144454956 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3493846356868744 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053127773106098175 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35586875677108765 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6541314721107483 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5870780348777771 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40729379653930664 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3827584683895111 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8027581572532654 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 *particle/sensitivity‚0.641911506652832 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6645923256874084 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.38849857449531555 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6316066384315491 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36056089401245117 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7050897479057312 +jet/accuracy_2_of_2‚0.17263178527355194 *jet/accuracy_2_of_3‚0.6049410700798035 *jet/accuracy_2_of_4‚0.8373957276344299 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3095727860927582 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2897402346134186 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 +jet/accuracy_3_of_3‚0.07885482162237167 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19163218140602112 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7172498106956482 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03082011267542839 *jet/accuracy_1_of_1‚0.3728956878185272 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1796320676803589 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03302428871393204 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7456220984458923 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9640940427780151 )loss/val_total_loss‚7.717541601751094 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2972675859928131 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2699529230594635 *jet/accuracy_1_of_2‚0.7912917137145996 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 7validation_average_jet_accuracy‚0.40683045983314514 trainer/global_step‚13719 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7942200303077698 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028090298175811768 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5164191722869873 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36926814913749695 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 particle/accuracy_1_of_1‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07369927316904068 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4590897262096405 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 particle/accuracy_2_of_2‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02649708092212677 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04384445399045944 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7160285711288452 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6916908621788025 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17894676327705383 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9832039475440979 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9130017161369324 _runtime‚24930.049623591 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3884442150592804 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453ø.æúL‘9‚(:Á  µ»‹ÓÞþé³ disk.drbd1931.in‚0 disk.drbd1828.out‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 952.546875 disk.drbd1289.in‚0# proc.memory.rssMB‚ 2923.18359375% disk./.usageGB‚1260.2687225341797 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 124.048828125# disk.dm-0.out‚57108.66162109375 disk.drbd1226.out‚0# memory_percent‚92.2367254892985 disk.sda.out‚ 952.546875& disk.nvme0n1.out‚57108.66162109375 disk.drbd1510.in‚0 cpu‚25.343308011171704 network.sent‚ 2.3992248e+07 disk.drbd1817.out‚0 disk.sda2.in‚0.046875 disk.drbd2041.in‚0 disk.drbd1550.out‚0 disk.drbd1817.in‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1510.out‚0 disk.drbd1938.out‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.drbd1226.in‚0* proc.memory.percent‚11.894464492797852) disk./.usagePercent‚70.48082440978123 disk.drbd2118.in‚0 disk.drbd1356.in‚0 network.recv‚ 1.760827e+06 disk.drbd1673.in‚0 disk.drbd1965.in‚0 disk.drbd1285.in‚0 disk.drbd1552.out‚0) proc.memory.availableMB‚ 1907.90234375 proc.cpu.threads‚17 disk.drbd2078.in‚0 disk.drbd1584.in‚0 disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.sda.in‚0.046875 disk.drbd1673.out‚0 disk.drbd1938.in‚0 disk.drbd2078.out‚0 disk.rbd1.in‚0 disk.drbd2118.out‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 144.8828125†.É1’9‚(:’ µ»‹Ó×Æ̳ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚280.156' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.38533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EŒòO“9‚(:Ä  Ä»‹ÓŧԳ network.sent‚ 2.3992314e+07 network.recv‚ 1.760893e+06 disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.sda.in‚0.046875 disk.sda2.in‚0.046875 disk.drbd1931.out‚0) disk./.usagePercent‚70.48082440978123 disk.drbd1356.out‚0% disk./.usageGB‚1260.2687225341797 disk.drbd1550.in‚0 disk.dm-0.in‚ 144.8828125 cpu‚25.34326237878057 disk.drbd1356.in‚0# proc.memory.rssMB‚ 2923.18359375 disk.drbd1289.out‚0 disk.drbd1965.in‚0 disk.drbd1510.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0 disk.drbd2078.in‚0 disk.drbd1552.in‚0 disk.sda2.out‚ 952.86328125 disk.drbd1931.in‚0 disk.drbd1938.in‚0$ memory_percent‚92.23769505818686 disk.drbd2041.in‚0 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.drbd1289.in‚0 proc.cpu.threads‚17 disk.drbd1817.out‚0# disk.dm-0.out‚57108.86083984375 disk.drbd2078.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.sda.out‚ 952.86328125 disk.drbd1938.out‚0 disk.drbd2111.in‚0 disk.rbd1.out‚ 124.298828125& disk.nvme0n1.out‚57108.86083984375 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd2118.out‚0 disk.drbd1965.out‚0 disk.drbd1226.out‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd2118.in‚0 disk.drbd1673.in‚0* proc.memory.percent‚11.894464492797852 disk.drbd1828.out‚0( proc.memory.availableMB‚ 1907.6640625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 144.8828125^Ë*ß“”9‚(:ˆ Ä»‹ÓÏ÷²³ gpu.0.gpu‚82.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚303.345' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚101.115 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÒÄ•9‚(:’ Ó»‹Ó“¢È³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚279.145' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.04833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BòP–9‚(:Å  Ó»‹ÓÐËî³ disk.drbd1285.in‚0 disk.drbd1931.in‚0 disk.sda.in‚0.046875 disk.sda.out‚ 953.13671875# proc.memory.rssMB‚ 2923.18359375 network.sent‚ 2.3992464e+07& disk.nvme0n1.out‚57109.16162109375 disk.drbd1584.out‚0 disk.nvme0n1.in‚ 144.8828125 disk.drbd1226.out‚0 disk.drbd1550.in‚0 disk.drbd1965.in‚0% disk./.usageGB‚1260.2686500549316 disk.drbd1817.in‚0 disk.sda2.out‚ 953.13671875 disk.drbd1938.in‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1356.in‚0# memory_percent‚92.2372023264567) disk./.usagePercent‚70.48082035636226 disk.drbd2078.in‚0 proc.cpu.threads‚17 disk.drbd1285.out‚0 disk.drbd1552.in‚0* proc.memory.percent‚11.894464492797852 network.recv‚ 1.761043e+06 disk.drbd1828.out‚0 disk.drbd1584.in‚0# disk.dm-0.out‚57109.16162109375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0 disk.drbd1673.out‚0 cpu‚25.343206022139082 disk.drbd1226.in‚0 disk.drbd1817.out‚0 disk.drbd1289.in‚0 disk.drbd1965.out‚0) proc.memory.availableMB‚ 1907.78515625 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd2078.out‚0 disk.drbd2041.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚ 124.392578125 disk.drbd1552.out‚0 disk.sda2.in‚0.046875 disk.drbd2111.in‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1289.out‚0 disk.drbd1510.in‚0 disk.drbd1938.out‚0 disk.drbd1356.out‚0±p~]4—9/ _wandbruntime‚24980  _runtime‚24980RBFʘ9Ä & lr-AdamW/pg1‚9.669943664556894e-08 & lr-AdamW/pg2‚9.669943664556894e-08 trainer/global_step‚13749 % _timestamp‚1.7848642173213053e+09 _runtime‚24980.678561108 _step‚1145-»2Ï™9É _runtime‚24980.678561108 _step‚1145 & lr-AdamW/pg1‚9.669943664556894e-08 & lr-AdamW/pg2‚9.669943664556894e-08 trainer/global_step‚13749 % _timestamp‚1.7848642173213053e+09ù1Âî4š9/ _wandbruntime‚24980  _runtime‚24980hú6x›9˜P KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _runtime‚24980.681628679 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3960563961796608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36642411 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1083155071199232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14353220613632 _step‚1146 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14353220613632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚311 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 410969600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3959002238977296 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3974915836974080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3973331428035152 trainer/global_step‚13749 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚316 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78572627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083154703108096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 % _timestamp‚1.7848642173224046e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17546118525440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14329189057856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14327844446570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3960563987970048 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14351875177472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3974915836974080 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3960563987970048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127711685 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3974915836974080 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3960563987970048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127711685 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42878464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3960563961796608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36642411 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1083155071199232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14353220613632 _step‚1146 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14353220613632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚311 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 410969600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3959002238977296 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3974915836974080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3973331428035152 trainer/global_step‚13749 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚316 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78572627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083154703108096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 % _timestamp‚1.7848642173224046e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17546118525440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0úZrúÿ49/ _wandbruntime‚24982  _runtime‚24982.Úz:''ž9¡N `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1083233289474048 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14328865248210 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1100781036669440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119441193 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41933174 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚321 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1100780623609856 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3959281015323104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083233657565184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3973609880571314 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41932858 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14354243128832 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3960842871087104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3960842871087104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36645051 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247151896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78577909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17547379104256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247161980 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3975197140389376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56PsXÞC#Ÿ9¦N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14352897685504 $ _timestamp‚1.784864218802537e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1083233289474048 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14328865248210 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1100781036669440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119441193 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41933174 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚321 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1100780623609856 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3959281015323104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083233657565184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3973609880571314 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41932858 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14354243128832 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3960842871087104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3960842871087104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36645051 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247151896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78577909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17547379104256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 BDeviceStatsMonitor.on_train_batch_endfsuŒé/allocation.all.allocated‚ 247161980 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3975197140389376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248008749 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14380292940664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 _step‚1150 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119851182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14403064733696 trainer/global_step‚13799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128157573 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3987321250567506 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128167637 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248018819 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42078221 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3972942327666096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3972942302238128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36774416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119851182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42078575 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14378948329378 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3988912491055104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14403064733696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128157573 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1087084021405696 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78852991 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78852632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755lý^‚´9›P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3988913862664704 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1104693256096768 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1104692834641408 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3987322620606760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128167637 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3974509452494848 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119851176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3988913862664704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119851182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42078575 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14378948329378 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3988912491055104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14403064733696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚359 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128157573 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1087084021405696 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78852991 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78852632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17608866599936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14404410169856 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3988912491055104 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3974509426321408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3974509452494848 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53364224 # _timestamp‚1.78486430381291e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248008749 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚354 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14380292940664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 _step‚  3Ÿ 1150 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119851182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14403064733696 trainer/global_step‚13799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128157573 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3987321250567506 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128167637 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248018819 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42078221 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3972942327666096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3972942302238128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36774416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248036282 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _step‚1151 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248026198 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3989193794470400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14405432685056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14404087241728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36777051 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248026198 _runtime‚25068.649458089 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78858236 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14405432685056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128176733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42081535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42081185 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3973221129439872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119859549 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3974788361785344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78858591 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17610116166656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1087162607771648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3987602443182288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248036282 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119859543 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3973221104011904 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 % _timestamp‚1.7848643052918253e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128176733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36777056ø¡¥¹-'·9§N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3987601073142922 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14381313742416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3974788361785344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78858591 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17610116166656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1087162607771648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3987602443182288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248036282 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119859543 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3973221104011904 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 % _timestamp‚1.7848643052918253e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128176733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36777056 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1104773092029440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14379969131018 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17610064906752 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 trainer/global_step‚13799 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3989193794470400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3974788361785344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248036282 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 419351040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 _step‚1151 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51259904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248026198 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3989193794470400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14405432685056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14404087241728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36777051 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248026198 _runtime‚25068.649458089 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78858236 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14405432685056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128176733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42081535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42081185 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3973221129439872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119859549 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 Forcent‚11.894464492797852 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.drbd1356.out‚0 disk.drbd1584.out‚0 disk.sda2.in‚0.046875 disk.drbd1289.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚57113.130859375 disk.dm-0.in‚ 144.8828125 disk.nvme0n1.in‚ 144.8828125 disk.drbd1226.out‚0 disk.drbd1510.in‚0 disk.drbd1938.in‚0³3¿PÃ9‚(:Å  ø¼‹Ó• Í³ disk.drbd1356.out‚0 disk.drbd2041.out‚0 disk.drbd2118.out‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.sda.in‚0.046875 disk.drbd2111.out‚0% disk.nvme0n1.out‚57113.4521484375 disk.drbd1965.in‚0 disk.drbd2078.in‚0 disk.drbd2041.in‚0 disk.drbd2118.in‚0 cpu‚25.342714807661924 disk.drbd1510.out‚0 disk.drbd1550.out‚0# proc.memory.rssMB‚ 2923.18359375 disk.drbd1673.out‚0 disk.sda2.out‚ 957.10546875" disk.dm-0.out‚57113.4521484375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.drbd1931.out‚0 disk.drbd1584.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 144.8828125 disk.sda.out‚ 957.10546875) proc.memory.availableMB‚ 1908.07421875 disk.sda2.in‚0.046875 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0$ memory_percent‚92.23602612813315* proc.memory.percent‚11.894464492797852 network.sent‚ 2.4206057e+07 disk.drbd1550.in‚0 disk.drbd1552.out‚0 disk.drbd1817.in‚0 disk.drbd2111.in‚0 disk.drbd1289.in‚0 disk.drbd2078.out‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0% disk./.usageGB‚1260.2628707885742 disk.drbd1673.in‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd1226.out‚0 disk.drbd1584.out‚0) disk./.usagePercent‚70.48049714953338 network.recv‚ 1.775251e+06 disk.dm-0.in‚ 144.8828125 disk.sda1.in‚0 disk.drbd1931.in‚0 disk.drbd1938.out‚0 disk.drbd1965.out‚0 disk.rbd1.out‚124.9814453125Kí¾¶Ä9‚(:’ ø¼‹ÓÒª³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚272.276' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.75866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉûÂ4Å9/ _wandbruntime‚25153  _runtime‚25153M&ì›ÊÆ9Ä & lr-AdamW/pg1‚3.500311381993009e-08 & lr-AdamW/pg2‚3.500311381993009e-08 trainer/global_step‚13849 % _timestamp‚1.7848643904591103e+09 _runtime‚25153.816349297 _step‚1153[®Ë¿ÏÇ9É & lr-AdamW/pg1‚3.500311381993009e-08 & lr-AdamW/pg2‚3.500311381993009e-08 trainer/global_step‚13849 % _timestamp‚1.7848643904591103e+09 _runtime‚25153.816349297 _step‚1153 IIsq4È9/ _wandbruntime‚25153  _runtime‚25153+^ëÇ(É9˜P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4001312443139114 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14430052212186 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120269526 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4002909145136128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42225697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79132486 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4001313813178368 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3988454890846208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17671064054272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4002910516745728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 _step‚1154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120269532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14454254289920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36906416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79132108 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4002909145136128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14454254289920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚373 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3988454917019648 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1091013339703296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14455599726080 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚25153.820938776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1108684403757568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7848643904602013e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128623589 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1091013707794432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17671123709952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248893121 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚378 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14431396823472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36906411 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120269526 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1108684831504384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ôd<#(Ê9P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4002910516745728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128623589 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3986882416354896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128613525 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4002909145136128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42225697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79132486 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4001313813178368 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3988454890846208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17671064054272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4002910516745728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 _step‚1154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120269532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14454254289920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36906416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79132108 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4002909145136128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14454254289920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚373 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3988454917019648 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1091013339703296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14455599726080 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚25153.820938776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1108684403757568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7848643904602013e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128623589 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1091013707794432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17671123709952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248893121 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚378 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14431396823472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36906411 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120269526 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1108684831504384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4001312443139114 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14430052212186 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120269526 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248910584 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1091092294160384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42228987 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4001593635753896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36909056 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17672301450240 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128632685 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14431073013826 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128622607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120277893 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120277893 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3988733826310144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128632685 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14456622241280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14455276797952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14456622241280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248910584 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128622607 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3988733800136704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79137668 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3988733800136704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42228617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4001592265714530 I@DeviceStatsMonitor.1**é; on_train_batch_end/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1091091926069248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17672356904448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120277899 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4003189076934656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 trainer/global_step‚13849 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248900500 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366C,n-'Ï9§N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120277899 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36909051 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3988733826310144 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1108764651064832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚370 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14432417625224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14455276797952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚25155.481814816 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4003190448551424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248910584 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1091092294160384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42228987 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4001593635753896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36909056 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 423545344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17672301450240 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128632685 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14431073013826 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128622607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120277893 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120277893 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3988733826310144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128632685 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55454208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14456622241280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14455276797952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14456622241280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248910584 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128622607 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3988733800136704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79137668 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3988733800136704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42228617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4001592265714530 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1091091926069248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17672356904448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120277899 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4003189076934656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 trainer/global_step‚13849 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248900500 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4000822505043696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79411489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14481156094994 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249757353 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129069477 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14505443846144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120687876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1094942658000896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4016905799217152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129079541 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1112676345504256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 trainer/global_step‚13899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14506789282304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120687882 UGDeviceStatsMonitor.on_trainâ²H]¹ _batch_start/allocation.large_pool.allocated‚ 120687882 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1112675917757440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42373078 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37038416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4016905799217152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4000822479615728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825&ÒÔ=#(â9P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120687876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1094942658000896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59655680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4016905799217152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129079541 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1112676345504256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 trainer/global_step‚13899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14506789282304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120687882 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120687882 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1112675917757440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42373078 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37038416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4016905799217152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4000822479615728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129069477 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4002400381544448 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14505443846144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4002400381544448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79411885 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚396 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129079541 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚391 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37038411 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17733319412224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17733259756544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249767423 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120687876 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4015305005749976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4016907170826752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4000822505043696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1730150400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79411489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14481156094994 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249757353 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129069477 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 427746816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47515172864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14505443846144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264† "i±H4ã9/ _wandbruntime‚25241  _runtime‚25241Š˜.Ù''ä9¡N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249774802 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79417044 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4017185731015680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚13899 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249784886 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4002679290834944 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1112755752096768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42375993 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249774802 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249784886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.784864478397107e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120696249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4002679264661504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120696249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120696243 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4017187102632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚393 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42376381 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4001101281389504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14507811797504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129088637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4015583458286138 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17734507729920 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129088637 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4002679264661504 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1112756179836416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4015584828325504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37041051 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 _runtime‚25241.754841099 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14506466354176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1095021612457984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181’ô„¦%å9¦N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17734507729920 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129088637 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4002679264661504 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1112756179836416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4015584828325504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37041051 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 _runtime‚25241.754841099 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14506466354176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1095021612457984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249774802 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79417044 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4017185731015680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚13899 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249784886 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4002679290834944 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1112755752096768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42375993 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249774802 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249784886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.784864478397107e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120696249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4002679264661504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120696249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120696243 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4017187102632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚393 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42376381 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4001101281389504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14507811797504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129088637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 T?DeviceStatsMonito´¯ï”†r.on_train_batch_end/requested_bytes.all.freed‚4015583458286138 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0‡ X–4æ9/ _wandbruntime‚25241  _runtime‚25241q·öç9‹ 2loss/SRqqt2/assignment_loss‚0.5340117663576807 /loss/FRt1/detection_loss‚0.5059621230783504 1loss/SRqqt2/detection_loss‚0.5638376397239766 0loss/FRt1/assignment_loss‚1.6063500173216643 0loss/FRt2/assignment_loss‚0.6082510450056621 3loss/FBt2/assignment_loss‚0.0048636552642070474 % _timestamp‚1.7848644784322605e+09 _runtime‚25241.789481791 _step‚1160 epoch‚49 4loss/FBt1/assignment_loss‚0.00017054265153770842 /loss/FRt2/detection_loss‚1.4351871427514458 1loss/SRqqt1/detection_loss‚0.8969864001306946 0loss/FBt1/detection_loss‚0.39474572325125334 /loss/FBt2/detection_loss‚0.5229353913759148 %loss/total_loss‚7.638683313979542 trainer/global_step‚13899 2loss/SRqqt1/assignment_loss‚0.5653818670671548±ÙöÐè9 0loss/FRt1/assignment_loss‚1.6063500173216643 0loss/FRt2/assignment_loss‚0.6082510450056621 3loss/FBt2/assignment_loss‚0.0048636552642070474 % _timestamp‚1.7848644784322605e+09 _runtime‚25241.789481791 _step‚1160 epoch‚49 4loss/FBt1/assignment_loss‚0.00017054265153770842 /loss/FRt2/detection_loss‚1.4351871427514458 1loss/SRqqt1/detection_loss‚0.8969864001306946 0loss/FBt1/detection_loss‚0.39474572325125334 /loss/FBt2/detection_loss‚0.5229353913759148 %loss/total_loss‚7.638683313979542 trainer/global_step‚13899 2loss/SRqqt1/assignment_loss‚0.5653818670671548 2loss/SRqqt2/assignment_loss‚0.5340117663576807 /loss/FRt1/detection_loss‚0.5059621230783504 1loss/SRqqt2/detection_loss‚0.5638376397239766ˆ ž!Ëré9‚(:’ ὋÓ³ÅÀ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚281.219' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.73966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 œe”Iê9‚(:¾  ὋÓ“¹à³ proc.memory.rssMB‚ 2923.1875 disk.drbd1289.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1907.12890625 cpu‚25.342448056250273 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.sda2.in‚0.046875 disk.drbd1226.out‚0 disk.drbd1938.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.sda2.out‚ 959.9296875 disk.drbd1356.out‚0% disk./.usageGB‚1260.2631378173828 disk.rbd1.in‚0% disk.nvme0n1.out‚57117.2158203125 disk.drbd1552.in‚0 disk.sda.in‚0.046875$ memory_percent‚92.23987261454263 disk.nvme0n1.in‚ 144.8828125 disk.drbd1965.out‚0 disk.drbd1550.in‚0 disk.drbd1817.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.drbd1552.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 125.591796875 disk.drbd1285.in‚0 disk.drbd1817.out‚0 disk.drbd1931.in‚0 network.recv‚ 1.783333e+06 disk.drbd2078.out‚0 disk.drbd1356.in‚0 disk.drbd1226.in‚0 disk.drbd1584.in‚0 disk.drbd1673.out‚0) disk./.usagePercent‚70.48051208318223 disk.dm-0.in‚ 144.8828125" disk.dm-0.out‚57117.2158203125 disk.drbd1965.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda.out‚ 959.9296875* proc.memory.percent‚11.894480387369793 proc.cpu.threads‚17 disk.drbd1938.out‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1584.out‚0 disk.drbd1828.out‚0 network.sent‚ 2.4322426e+07 disk.drbd2041.in‚0kÚøµGë9‚(:¼  ð½‹ÓèÔé³* proc.memory.percent‚11.894480387369793 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.drbd1965.out‚0& disk.nvme0n1.out‚57117.56591796875 disk.drbd1550.out‚0 disk.sda2.in‚0.046875 disk.drbd1931.in‚0 disk.drbd1289.in‚0 network.sent‚ 2.4322534e+07 network.recv‚ 1.783441e+06 disk.drbd1938.out‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd1817.in‚0 disk.drbd1931.out‚0# disk.dm-0.out‚57117.56591796875 disk.drbd2078.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd2041.in‚0 disk.drbd1510.in‚0$ memory_percent‚92.24087397257486 disk.drbd2118.out‚0 disk.drbd1965.in‚0( proc.memory.availableMB‚ 1906.8828125# disk./.usageGB‚1260.26314163208 disk.drbd1552.in‚0 disk.sda.out‚ 960.203125 disk.drbd1356.out‚0 proc.cpu.threads‚17 disk.drbd1356.in‚0 disk.rbd1.out‚125.6240234375 disk.drbd1510.out‚0 disk.drbd2118.in‚0 disk.drbd1938.in‚0 disk.drbd1828.out‚0 disk.rbd1.in‚0 disk.drbd1584.in‚0 disk.sda.in‚0.046875 disk.sda2.out‚ 960.203125 cpu‚25.342432615960245 disk.drbd1673.out‚0 disk.drbd2078.out‚0 disk.drbd2041.out‚0 disk.drbd1226.in‚0 proc.memory.rssMB‚ 2923.1875 disk.drbd1552.out‚0 disk.drbd2111.in‚0 disk.drbd1289.out‚0 disk.drbd1584.out‚0 disk.drbd1550.in‚0) disk./.usagePercent‚70.48051229652008 disk.drbd1673.in‚0 disk.drbd2111.out‚0 disk.drbd1828.in‚0 disk.nvme0n1.in‚ 144.8828125á€9Jì9‚(:’ ð½‹ÓÞÒɳ gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚296.543' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚98.84766666666667 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lk.Xží9‚(:“ ÿ½‹ÓäÀµ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚61.0 gpu.0.powerWatts‚107.649' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚35.882999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú|ëCî9‚(:¸  ÿ½‹Ó™¸Ó³ disk.drbd1550.in‚0 disk.drbd1938.in‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.nvme0n1.in‚ 144.8828125 disk.sda1.out‚0 disk.drbd1965.out‚0 disk.drbd1965.in‚0 network.recv‚ 1.783507e+06 disk.drbd1584.out‚0 disk.sda.in‚0.046875 disk.drbd1931.out‚0 disk.drbd2078.in‚0 disk.drbd1828.in‚0 disk.drbd1226.in‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1226.out‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚57117.96435546875 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd2118.in‚0 disk.drbd2078.out‚0 disk.drbd1356.in‚0 proc.memory.rssMB‚ 2923.1875 disk.drbd1510.in‚0 disk.drbd1673.out‚0 disk.drbd1289.in‚0 disk.drbd1828.out‚0 disk.rbd1.out‚125.8115234375 disk.drbd2118.out‚0 disk.drbd1938.out‚0# disk.dm-0.out‚57117.96435546875 disk.drbd2041.in‚0( proc.memory.availableMB‚ 1906.8828125 network.sent‚ 2.43226e+07 disk.drbd1817.in‚0 disk.sda2.in‚0.046875 disk.sda2.out‚960.4375 disk.drbd1931.in‚0$ memory_percent‚92.24087397257486 disk.drbd2041.out‚0* proc.memory.percent‚11.894480387369793 cpu‚25.342378199268467% disk./.usageGB‚1260.2635536193848 disk.drbd1552.out‚0 disk.dm-0.in‚ 144.8828125 proc.cpu.threads‚17 disk.drbd1552.in‚0 disk.sda1.in‚0 disk.drbd1817.out‚0 disk.sda.out‚960.4375) disk./.usagePercent‚70.48053533700688 disk.drbd1673.in‚0Í_VH’ï9‚(:‡ Ž¾‹Ó½§³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚281.991' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òÉ)DJð9‚(:¿  Ž¾‹Ó‰ŠÏ³% disk./.usageGB‚1260.2635536193848 disk.drbd1931.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.48053533700688 disk.nvme0n1.in‚ 144.8828125 network.recv‚ 1.787108e+06 disk.drbd1285.in‚0 disk.drbd1550.in‚0 disk.sda2.in‚0.046875 disk.drbd1289.in‚0 disk.drbd1938.in‚0* proc.memory.percent‚11.894480387369793 disk.drbd1226.out‚0 disk.drbd1550.out‚0 disk.drbd1356.in‚0$ memory_percent‚92.24158922831217 disk.drbd1552.in‚0 disk.drbd1931.out‚0 disk.drbd2041.in‚0 disk.drbd2078.in‚0 disk.drbd1226.in‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1552.out‚0 disk.drbd2118.out‚0 disk.drbd1673.in‚0! disk.dm-0.out‚57118.533203125 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.rbd1.out‚125.8740234375 disk.drbd1510.out‚0 disk.sda.in‚0.046875 disk.sda2.out‚ 960.87890625 disk.drbd1938.out‚0 disk.drbd2111.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1356.out‚0 disk.drbd1817.in‚0) proc.memory.availableMB‚ 1906.70703125 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.drbd1289.out‚0 disk.drbd2078.out‚0 proc.memory.rssMB‚ 2923.1875 disk.drbd2111.in‚0 disk.drbd1510.in‚0 disk.drbd1584.out‚0 disk.sda.out‚ 960.87890625 disk.drbd2118.in‚0 disk.drbd1285.out‚0$ disk.nvme0n1.out‚57118.533203125 disk.drbd1828.in‚0 disk.drbd1828.out‚0 disk.drbd1817.out‚0 disk.drbd1965.in‚0 disk.drbd1584.in‚0 cpu‚25.342343293685506 network.sent‚ 2.4424039e+07åÞï¨ñ9‚(:’ ¾‹ÓæðÒ³ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.842' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.28066666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ?8(Gò9‚(:¼  ¾‹ÓšÄð³ disk.drbd2041.out‚0 disk.drbd1510.out‚0 disk.drbd1673.in‚0 disk.sda2.in‚0.046875% disk./.usageGB‚1260.2635536193848 disk.rbd1.out‚125.8740234375 disk.drbd1550.out‚0 disk.drbd1285.in‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1965.in‚0 disk.drbd1828.out‚0 disk.drbd1510.in‚0 disk.sda.in‚0.046875 network.sent‚ 2.4425003e+07! disk.dm-0.out‚57118.732421875 disk.drbd1285.out‚0 disk.sda.out‚ 961.2890625 disk.drbd1938.in‚0 disk.drbd1938.out‚0$ disk.nvme0n1.out‚57118.732421875 disk.drbd2111.out‚0 disk.drbd1931.out‚0* proc.memory.percent‚11.894480387369793 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.drbd1552.out‚0 disk.drbd2118.in‚0 cpu‚25.34231768067281 disk.drbd1965.out‚0 disk.drbd2078.out‚0 disk.drbd1356.out‚0 disk.drbd1673.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0) proc.memory.availableMB‚ 1906.66015625 disk.nvme0n1.in‚ 144.8828125 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.sda1.out‚0 disk.drbd1817.out‚0 disk.sda2.out‚ 961.2890625 proc.memory.rssMB‚ 2923.1875 disk.drbd1356.in‚0) disk./.usagePercent‚70.48053533700688 disk.drbd2118.out‚0 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1226.in‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0$ memory_percent‚92.24177996317545 proc.cpu.threads‚17 network.recv‚ 1.788004e+06§¬Èó9‚(:’ ¬¾‹Ó’ɳ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚290.226' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.74199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H/OþAô9‚(:¶  ¬¾‹ÓÊ•ä³ disk.drbd1817.in‚0& proc.memory.availableMB‚ 1906.90625 disk.rbd1.in‚0 disk.drbd1285.in‚0 disk.drbd1510.in‚0 disk.drbd1550.in‚0$ memory_percent‚92.24077860514322 network.recv‚ 1.788112e+06 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1828.in‚0 disk.drbd1226.out‚0 disk.drbd1938.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.rbd1.out‚ 125.90625 disk.drbd1510.out‚0 cpu‚25.34227312231131% disk./.usageGB‚1260.2635536193848 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd1356.out‚0 disk.drbd1289.in‚0* proc.memory.percent‚11.894480387369793 disk.drbd1817.out‚0) disk./.usagePercent‚70.48053533700688 disk.drbd2118.out‚0 proc.memory.rssMB‚ 2923.1875 network.sent‚ 2.4425111e+07 disk.drbd1584.out‚0 disk.drbd1673.in‚0 disk.dm-0.in‚ 144.8828125" disk.dm-0.out‚57119.1416015625 disk.drbd1356.in‚0 proc.cpu.threads‚17 disk.drbd1931.out‚0 disk.drbd1938.in‚0 disk.nvme0n1.in‚ 144.8828125 disk.drbd1552.out‚0 disk.drbd1965.in‚0 disk.drbd2111.out‚0 disk.drbd1550.out‚0 disk.drbd1552.in‚0 disk.sda.out‚ 961.5390625 disk.sda2.out‚ 961.5390625 disk.drbd1584.in‚0 disk.sda2.in‚0.046875% disk.nvme0n1.out‚57119.1416015625 disk.sda1.out‚0 disk.drbd2078.out‚0 disk.drbd1285.out‚0 disk.drbd1226.in‚0 disk.sda1.in‚0 disk.sda.in‚0.046875 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.out‚0f–-4õ9/ _wandbruntime‚25327  _runtime‚25327·à•Êö9Ä & lr-AdamW/pg2‚3.993295684889883e-09 trainer/global_step‚13949 % _timestamp‚1.7848645638274047e+09 _runtime‚25327.185900559 _step‚1161 & lr-AdamW/pg1‚3.993295684889883e-09öÿ¨Ï÷9É _runtime‚25327.185900559 _step‚1161 & lr-AdamW/pg1‚3.993295684889883e-09 & lr-AdamW/pg2‚3.993295684889883e-09 trainer/global_step‚13949 % _timestamp‚1.7848645638274047e+09‰ æ$qž4ø9/ _wandbruntime‚25327  _runtime‚25327wÐ`W(ù9—P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1116667350272000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4014762568304528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14556633402368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1098872344389632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129525429 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚1162 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79691630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250641725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79691241 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1116667775921664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4030902453298176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14557978838528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4030902453298176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4029294828282330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121106226 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42521214 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚13949 _runtime‚25327.18600877 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400lÝú9œP M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129525429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42520830 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4030903824907776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4014762568304528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14556633402368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1098872344389632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129525429 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47515172864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1181 _step‚1162 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79691630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57558528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250641725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79691241 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1116667775921664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4030902453298176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14557978838528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4030902453298176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4029294828282330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121106226 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42521214 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚13949 _runtime‚25327.18600877 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1116667350272000 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129534511 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14557655910400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129544589 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79696842 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37173051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 trainer/global_step‚13949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14534625390840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121114593 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79697234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42524178 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1098950562664448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129544589 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚387 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1116747607197184 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14533280779442 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250659188 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17796676441600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42523791 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4029576020897112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4031183756713472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚25328.658637407 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4031183756713472 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121114599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1116747177360384 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1098950930755584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250649104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14557655910400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4016624729186304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250649104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7848645653010178e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121114593 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17796614695936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121114599 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ù€eý-'ý9§N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250649104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7848645653010178e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121114593 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17796614695936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121114599 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37173056 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129534511 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14557655910400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129544589 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79696842 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37173051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61745664 trainer/global_step‚13949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14534625390840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121114593 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79697234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42524178 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49245323264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1098950562664448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129544589 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚387 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1116747607197184 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14533280779442 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250659188 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17796676441600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47515172864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42523791 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46288510976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 429836800 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4029576020897112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4031183756713472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚25328.658637407 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4031183756713472 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121114599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1116747177360384 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1098950930755584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2006 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250649104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14557655910400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4016624729186304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264‹ ð•Î“4þ9/ _wandbruntime‚25328  _runtime‚25328|/OÍÿ9ˆ 2loss/SRqqt1/assignment_loss‚0.4671621962149166 2loss/FBt2/assignment_loss‚0.009865788618702255 1loss/SRqqt2/detection_loss‚0.5805098723621415 /loss/FBt1/detection_loss‚0.7307424688055604 3loss/SRqqt2/assignment_loss‚0.47369920893993095 epoch‚49 _runtime‚25328.695639559 _step‚1164 3loss/FBt1/assignment_loss‚8.912869370149248e-05 /loss/FRt1/detection_loss‚0.5528267341754394 1loss/SRqqt1/detection_loss‚0.9815744591571721 /loss/FBt2/detection_loss‚0.3582727118036928 trainer/global_step‚13949 0loss/FRt1/assignment_loss‚1.4797253548980704 /loss/FRt2/detection_loss‚1.3025160852004265 $loss/total_loss‚7.52398494006401 % _timestamp‚1.7848645653384345e+09 0loss/FRt2/assignment_loss‚0.5870009311942554ÒÆg€: 0loss/FRt1/assignment_loss‚1.4797253548980704 /loss/FRt2/detection_loss‚1.3025160852004265 $loss/total_loss‚7.52398494006401 % _timestamp‚1.7848645653384345e+09 0loss/FRt2/assignment_loss‚0.5870009311942554 2loss/SRqqt1/assignment_loss‚0.4671621962149166 2loss/FBt2/assignment_loss‚0.009865788618702255 1loss/SRqqt2/detection_loss‚0.5805098723621415 /loss/FBt1/detection_loss‚0.7307424688055604 3loss/SRqqt2/assignment_loss‚0.47369920893993095 epoch‚49 _runtime‚25328.695639559 _step‚1164 3loss/FBt1/assignment_loss‚8.912869370149248e-05 /loss/FRt1/detection_loss‚0.5528267341754394 1loss/SRqqt1/detection_loss‚0.9815744591571721 /loss/FBt2/detection_loss‚0.3582727118036928 trainer/global_step‚13949Œ ª³¤<’:‚(:‡ »¾‹Ó¹¾Û³ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚296.853' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚98.951 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£gjdG‚:‚(:¼  »¾‹Óı´ disk.drbd1510.out‚0 disk.drbd2078.in‚0 disk.drbd1356.out‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.sda.in‚0.046875 disk.sda2.out‚ 962.0546875 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1938.in‚0) disk./.usagePercent‚70.48053533700688 disk.drbd1550.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0" disk.dm-0.out‚57119.3408203125 network.recv‚ 1.791821e+06 disk.drbd1550.out‚0 disk.drbd2111.in‚0 disk.drbd1226.out‚0% disk./.usageGB‚1260.2635536193848 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd1552.in‚0$ memory_percent‚92.24104881286621 proc.cpu.threads‚17% disk.nvme0n1.out‚57119.3408203125 disk.drbd2118.in‚0 disk.drbd1938.out‚0 disk.sda.out‚ 962.0546875 disk.drbd1931.out‚0 disk.drbd2111.out‚0 disk.rbd1.out‚ 126.0078125 disk.nvme0n1.in‚ 144.8828125 disk.drbd1552.out‚0 disk.drbd1817.out‚0 disk.sda2.in‚0.046875 cpu‚25.342237802617955 proc.memory.rssMB‚ 2923.1875 disk.drbd1584.out‚0 disk.drbd1289.out‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1965.out‚0 disk.drbd2078.out‚0) proc.memory.availableMB‚ 1906.83984375 disk.drbd1285.out‚0 disk.drbd1817.in‚0 network.sent‚ 2.4490507e+07 disk.drbd1584.in‚0 disk.drbd1965.in‚0 disk.drbd1356.in‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1828.in‚0 disk.rbd1.in‚0 disk.drbd1673.out‚0 disk.drbd1931.in‚0* proc.memory.percent‚11.894480387369793 disk.drbd1226.in‚08¶g7žƒ:‚(:“ ʾ‹Óóäè³ gpu.0.gpu‚67.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚306.242' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚102.08066666666666 gpu.0.smClock‚1350 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žU9‹D„:‚(:¹  ʾ‹Óù鈴 disk.drbd1817.out‚0 disk.sda2.in‚0.046875! disk.dm-0.out‚57119.619140625 disk.drbd1285.in‚0 proc.memory.rssMB‚ 2923.1875 network.sent‚ 2.4491614e+07 disk.drbd1552.in‚0 disk.drbd1289.out‚0 disk.drbd1552.out‚0 disk.sda.out‚ 962.37109375$ memory_percent‚92.24287668863931 proc.cpu.threads‚17 network.recv‚ 1.792789e+06 disk.drbd2041.in‚0) disk./.usagePercent‚70.48053256361496 disk.drbd1584.out‚0 disk.rbd1.out‚ 126.0703125 disk.drbd2118.in‚0 disk.drbd1673.out‚0 disk.drbd2078.out‚0$ disk.nvme0n1.out‚57119.619140625 disk.drbd1931.in‚0 disk.drbd1938.out‚0' proc.memory.availableMB‚ 1906.390625 disk.drbd1510.out‚0 disk.sda1.out‚0 disk.drbd2118.out‚0 disk.drbd1965.in‚0 disk.drbd1226.out‚0 disk.sda2.out‚ 962.37109375 disk.dm-0.in‚ 144.8828125 disk.drbd1356.in‚0 disk.drbd1356.out‚0% disk./.usageGB‚1260.2635040283203 disk.drbd1550.in‚0 disk.sda1.in‚0 disk.drbd1931.out‚0 disk.nvme0n1.in‚ 144.8828125 disk.drbd1938.in‚0 disk.drbd1226.in‚0 cpu‚25.34221272052353 disk.drbd2111.out‚0 disk.drbd1285.out‚0 disk.drbd1510.in‚0 disk.drbd2078.in‚0 disk.drbd2111.in‚0 disk.drbd1828.in‚0 disk.drbd1289.in‚0 disk.drbd1550.out‚0 disk.drbd1584.in‚0 disk.drbd1817.in‚0 disk.drbd1673.in‚0 disk.sda.in‚0.046875 disk.drbd1828.out‚0* proc.memory.percent‚11.894480387369793 disk.drbd1965.out‚0 disk.drbd2041.out‚0 disk.rbd1.in‚0X„Û¦’…:‚(:‡ Ù¾‹Ó˜Ï«³ gpu.0.gpu‚99.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚65.0 gpu.0.powerWatts‚288.153' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.051 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žœÕ¾D†:‚(:¹  Ù¾‹Ó’¦Í³ disk.sda1.out‚0$ memory_percent‚92.24287668863931 disk.drbd1552.in‚0 network.sent‚ 2.4494064e+07 disk.rbd1.out‚ 126.0703125 disk.drbd1550.in‚0 disk.drbd1289.out‚0! disk.dm-0.out‚57119.857421875 disk.drbd1965.in‚0 disk.drbd1226.in‚0 disk.drbd1673.in‚0 disk.drbd1289.in‚0 disk.dm-0.in‚ 144.8828125 disk.sda.in‚0.046875 network.recv‚ 1.793741e+06 disk.drbd1584.in‚0 disk.sda2.out‚ 962.81640625 disk.drbd1965.out‚0 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd1510.in‚0 disk.drbd1817.in‚0 disk.sda2.in‚0.046875 disk.drbd2041.out‚0 disk.drbd1828.out‚0 disk.drbd2118.in‚0 disk.drbd1356.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 144.8828125 disk.drbd1673.out‚0 disk.drbd2111.in‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1931.in‚0' proc.memory.availableMB‚ 1906.390625* proc.memory.percent‚11.894480387369793 disk.drbd1931.out‚0 cpu‚25.342169069777842 disk.drbd2118.out‚0$ disk.nvme0n1.out‚57119.857421875 disk.drbd1510.out‚0 disk.drbd1550.out‚0 disk.drbd1938.in‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 proc.cpu.threads‚17 disk.drbd1285.in‚0 disk.drbd1285.out‚0 disk.drbd1226.out‚0 disk.sda.out‚ 962.81640625 disk.drbd1938.out‚0) disk./.usagePercent‚70.48053149692575$ disk./.usageGB‚1260.263484954834 disk.drbd1584.out‚0 disk.drbd1817.out‚0 proc.memory.rssMB‚ 2923.1875 disk.drbd1828.in‚0 disk.sda1.in‚0¹0ãÀ‡:‚(:’ 辋ÓÊų gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚64.0 gpu.0.powerWatts‚289.372' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.45733333333334 gpu.0.smClock‚1395 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œvíDˆ:‚(:¹  辋Óô½ç³ disk.sda2.in‚0.046875 disk.drbd2078.out‚0$ disk.nvme0n1.out‚57120.130859375 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1552.in‚0 disk.drbd1289.in‚0) disk./.usagePercent‚70.48053149692575 disk.nvme0n1.in‚ 144.8828125 disk.drbd1550.in‚0 disk.drbd2118.out‚0 disk.drbd2078.in‚0 disk.drbd1931.in‚0 cpu‚25.34215359085387 network.recv‚ 1.793981e+06 disk.drbd1285.out‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.drbd1817.in‚0* proc.memory.percent‚11.894480387369793$ memory_percent‚92.24287668863931 proc.memory.rssMB‚ 2923.1875 disk.drbd1510.out‚0 disk.sda1.in‚0 disk.drbd1673.in‚0 disk.drbd1828.in‚0 disk.rbd1.out‚126.1025390625 disk.drbd1584.out‚0 disk.sda.out‚ 963.1328125 disk.drbd1673.out‚0 disk.drbd1226.out‚0 disk.drbd1584.in‚0 disk.sda2.out‚ 963.1328125 disk.drbd1965.in‚0 disk.sda.in‚0.046875! disk.dm-0.out‚57120.130859375 disk.drbd1356.out‚0 disk.drbd2041.in‚0 disk.drbd2041.out‚0 disk.drbd2111.out‚0 disk.drbd1552.out‚0 disk.drbd1938.in‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1965.out‚0 network.sent‚ 2.4499472e+07 disk.drbd2111.in‚0 disk.drbd1828.out‚0 disk.drbd1285.in‚0 disk.drbd1550.out‚0 disk.drbd1817.out‚0 disk.drbd1289.out‚0' proc.memory.availableMB‚ 1906.390625 proc.cpu.threads‚17$ disk./.usageGB‚1260.263484954834 disk.rbd1.in‚0 disk.drbd1938.out‚0 disk.drbd1356.in‚0 disk.drbd1931.out‚0ž›¹<‰:‚(:’ ÷¾‹ÓÃú´³ gpu.0.gpu‚86.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚62.0 gpu.0.powerWatts‚109.045' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.34833333333333 gpu.0.smClock‚1410 ÎìS)agpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ÏC%@Š:‚(:µ  ÷¾‹ÓÒýÕ³ disk.drbd1828.out‚0 disk.drbd1584.in‚0 disk.drbd1938.in‚0 disk.rbd1.out‚126.2041015625 disk.drbd2078.in‚0 proc.cpu.threads‚17 proc.memory.rssMB‚ 2923.1875* proc.memory.percent‚11.894480387369793 network.sent‚ 2.4506486e+07) disk./.usagePercent‚70.48048754933053 disk.drbd1828.in‚0 disk.drbd1285.in‚0$ disk.nvme0n1.out‚57120.294921875 disk.drbd2118.out‚0 disk.sda.in‚0.046875# memory_percent‚92.2429084777832( proc.memory.availableMB‚ 1906.3828125 disk.drbd2111.in‚0 disk.sda1.out‚0 disk.drbd2118.in‚0 disk.sda2.in‚0.046875 disk.drbd2041.out‚0 disk.drbd1552.in‚0 disk.drbd1817.out‚0! disk.dm-0.out‚57120.294921875 disk.drbd1552.out‚0 disk.drbd1584.out‚0 disk.drbd1817.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 cpu‚25.342099517018568 network.recv‚ 1.803396e+06 disk.nvme0n1.in‚ 144.8828125 disk.drbd1226.out‚0 disk.drbd1673.in‚0 disk.drbd1931.out‚0 disk.drbd2078.out‚0% disk./.usageGB‚1260.2626991271973 disk.drbd2111.out‚0 disk.drbd1938.out‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1226.in‚0 disk.drbd1510.in‚0 disk.drbd1510.out‚0 disk.drbd1289.out‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.drbd2041.in‚0 disk.sda1.in‚0 disk.sda2.out‚963.4375 disk.drbd1931.in‚0 disk.drbd1356.out‚0 disk.drbd1550.in‚0 disk.rbd1.in‚0 disk.drbd1285.out‚0 disk.drbd1550.out‚0 disk.sda.out‚963.4375 disk.drbd1673.out‚0#:³‹:‚(:’ †¿‹Ó®äÛ³ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.96974182128907* gpu.0.memoryAllocatedBytes‚ 49795629056 gpu.0.temp‚66.0 gpu.0.powerWatts‚277.703' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.56766666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0søëeJŒ:‚(:¿  †¿‹Ó·«ü³$ memory_percent‚92.24316279093424 network.recv‚ 1.80424e+06 disk.drbd1817.out‚0 disk.drbd1673.in‚0" disk.dm-0.out‚57121.1669921875 disk.drbd2041.out‚0 disk.dm-0.in‚ 144.8828125 disk.drbd1550.in‚0 disk.drbd1552.in‚0 disk.drbd1817.in‚0 disk.sda.in‚0.046875 disk.drbd1931.out‚0 disk.drbd1285.out‚0 disk.drbd2118.out‚0 disk.drbd1938.in‚0 disk.drbd1356.out‚0% disk.nvme0n1.out‚57121.1669921875* proc.memory.percent‚11.894480387369793 disk.sda1.out‚0 disk.drbd2111.in‚0 proc.cpu.threads‚17 disk.drbd1673.out‚0 disk.sda2.in‚0.046875 disk.drbd1938.out‚0 disk.rbd1.in‚0 cpu‚25.342064242656484( proc.memory.availableMB‚ 1906.3203125% disk./.usageGB‚1260.2629280090332 disk.sda2.out‚ 963.88671875 disk.drbd1828.out‚0) disk./.usagePercent‚70.48050034960099 disk.drbd1931.in‚0 disk.drbd1289.out‚0 disk.drbd1510.out‚0 disk.drbd1550.out‚0 proc.memory.rssMB‚ 2923.1875 disk.drbd2118.in‚0 disk.sda.out‚ 963.88671875 disk.drbd2078.in‚0 disk.drbd2078.out‚0 disk.drbd2111.out‚0 disk.drbd1510.in‚0 disk.sda1.in‚0 disk.drbd1584.out‚0 disk.drbd1552.out‚0 disk.drbd1226.in‚0 disk.drbd1828.in‚0 disk.drbd1285.in‚0 disk.drbd1226.out‚0 disk.drbd2041.in‚0 disk.rbd1.out‚126.2666015625 network.sent‚ 2.4509475e+07 disk.drbd1584.in‚0 disk.drbd1289.in‚0 disk.drbd1965.in‚0 disk.drbd1965.out‚0 disk.drbd1356.in‚0 disk.nvme0n1.in‚ 144.8828125Œ€4:/ _wandbruntime‚25413  _runtime‚25413+B'~ÊŽ:Ä _step‚1165 & lr-AdamW/pg1‚1.535309901949749e-12 & lr-AdamW/pg2‚1.535309901949749e-12 trainer/global_step‚13999 % _timestamp‚1.7848646505553806e+09 _runtime‚25413.912608226p¢_Ï:É & lr-AdamW/pg1‚1.535309901949749e-12 & lr-AdamW/pg2‚1.535309901949749e-12 trainer/global_step‚13999 % _timestamp‚1.7848646505553806e+09 _runtime‚25413.912608226 _step‚1165 y?&4:/ _wandbruntime‚25413  _runtime‚254135‘¥(‘:˜P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12299 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344611286 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345436160 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4044900478988800 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4044899107379200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4043286020853938 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14607822958592 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121524582 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129981381 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79971164 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129991445 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37302411 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4030291310594048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4028702682421296 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚405 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14583363860610 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚1166 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4030291284420608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1120659263464960 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1120658831523840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚25413.916908487 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14584708471896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1102801662687232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251516027 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42668753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚63849984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251505957 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14609168394752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129991445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42669153 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1102801294596096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4028702656993328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129981381 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 431941120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17857600777728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4044900478988800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17857536927744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4044899107379200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 % _timestamp‚1.7848646505564053e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14609168394752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4030291310594048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121524576 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121524582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚13999 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14607822958592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49245323264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚405 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47709852672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14583363860610 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1181 _step‚1166 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4030291284420608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47709852672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚56 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1120659263464960 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1120658831523840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚25413.916908487 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46288510976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14584708471896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1653 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1102801662687232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679332050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49245323264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251516027 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42668753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚63849984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251505957 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12299 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14609168394752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129991445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42669153 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1102801294596096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1123180544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47667565755 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10070 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚825 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345436160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4028702656993328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47515172864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129981381 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 431941120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17857600777728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46288510976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4044900478988800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1730150400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371609600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚628 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17857536927744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1181 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚825 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12224 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4044899107379200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680172544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚825 % _timestamp‚1.7848646505564053e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14609168394752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚660 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370039254 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49245323264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1730150400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10070 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4030291310594048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121524576 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121524582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1653 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46246678528 trainer/global_step‚13999 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14607822958592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1410488832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350627328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4044899107379200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4043286020853938 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680172544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14607822958592 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121524582 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129981381 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79971164 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129991445 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371609600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37302411 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4030291310594048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4028702682421296 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14608845466624 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4045179039177728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37305056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚390 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚25415.38683444 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17858772965376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130000541 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251533490 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1167 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251533490 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129990463 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1102880249053184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14585729273648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14610190909952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4030570193711104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79976768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4030570219884544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121532949 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129990463 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4030570219884544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79977163 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4043567213468720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121532949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4045180410794496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1120738653927424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42672107 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14584384662250 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121532943 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251523406 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14608845466624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37305051 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚13999 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1120739081667072 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251523406 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4030570193711104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 % _timestamp‚1.7848646520292258e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4043565843429354 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4028981458767104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47709852672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4030570193711104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1181 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚628 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344611398 % _timestamp‚1.7848646520292258e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59648512 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4043565843429354 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10084 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371616768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4028981458767104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47515172864 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47709852672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14608845466624 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4045179039177728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46288510976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46246678528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37305056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚390 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚25415.38683444 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17858772965376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130000541 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251533490 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1123180544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1167 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251533490 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129990463 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680172544 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1102880249053184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚56 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14585729273648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47515172864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1730150400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14610190909952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345443328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350627328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1730150400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4030570193711104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79976768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47667565755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12299 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4030570219884544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121532949 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129990463 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4030570219884544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79977163 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4043567213468720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49245323264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121532949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4045180410794496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1120738653927424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42672107 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2006 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1410488832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14584384662250 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121532943 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680172544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251523406 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1653 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679332050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1730150400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370039366 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14608845466624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37305051 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2006 trainer/global_step‚13999 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371616768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 427739648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49245323264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345443328 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1120739081667072 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251523406 F&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3715417981147766 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 _step‚1169 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3903312683105469 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 *jet/accuracy_1_of_3‚0.9387781023979187 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 epoch‚49 % _timestamp‚1.7848646717154095e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7218963503837585 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.3000674247741699 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5463929772377014 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0314369834959507 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2700788676738739 particle/accuracy_3_of_4‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025345008820295334 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.28106024861335754 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04384445399045944 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35776278376579285 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7456220984458923 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9215782284736633 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6632474064826965 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6539655327796936 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6926869750022888 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.29060623049736023 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5282303690910339 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7003664970397949 *jet/accuracy_1_of_4‚0.9832039475440979 )loss/val_total_loss‚7.720648282247729 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4070996642112732 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4066423773765564 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7212480902671814 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.18073248863220215 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22555819153785706 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.27119913697242737 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6936321258544922 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_4‚0.10971427708864212 +particle/specificity‚0.9129796028137207 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7199500799179077 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5870780348777771 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17447948455810547 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2968718111515045 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34917065501213074 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3697308897972107 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6324979066848755 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658390998840332 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38531824946403503 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6772319674491882 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_2‚0.1727919578552246 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36255887150764465 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3582589030265808 7"Purity/0FRt*SRqqt1FBt/event_purity‚0.87553870677948 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053127773106098175 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17858301103115082 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7172498106956482 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7912930250167847 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7769578695297241 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 (particle/accuracy‚0.8548176884651184 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7956777811050415 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6383249163627625 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8373957276344299 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5180065035820007 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27085497975349426 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6292827725410461 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6548104882240295 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3882982134819031 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09070056676864624 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 particle/accuracy_2_of_3‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026956509798765182 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0745496079325676 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2793044149875641 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31152063608169556 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029196735471487045 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6436420679092407 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6915340423583984 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4590897262096405}16Š:•v 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025345008820295334 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9984424710273743 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.28106024861335754 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt*SRqqt2FBt/event_proportion‚0.01203264482319355 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9981260895729065 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988095164299011 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0032087054569274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04384445399045944 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2273298054933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.35776278376579285 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7456220984458923 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993947148323059 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9215782284736633 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6632474064826965 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6539655327796936 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6926869750022888 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.29060623049736023 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3796038031578064 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5282303690910339 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8608665466308594 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.466594934463501 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.3155677616596222 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7003664970397949 *jet/accuracy_1_of_4‚0.9832039475440979 )loss/val_total_loss‚7.720648282247729 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2045200914144516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 ,&PuritvÕ8I‘4y/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6340680718421936 9&Purity/2FRt*SRqqt0FBt/event_proportion‚0.039306640625 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630643486976624 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0416666679084301 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0377022884786129 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4070996642112732 6&Purity/*FRt0SRqqt1FBt/event_proportion‚ 0.041015625 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4066423773765564 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991288781166077 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6642427444458008 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7212480902671814 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9981332421302795 & Purity/1FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt1FBt/event_proportion‚0.005824497900903225 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.18073248863220215 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8780909776687622 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22555819153785706 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.27119913697242737 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85308438539505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6444153189659119 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00045340400538407266 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6936321258544922 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0009416852844879031 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006905692163854837 +jet/accuracy_3_of_4‚0.10971427708864212 +particle/specificity‚0.9129796028137207 9&Purity/0FRt*SRqqt*FBt/event_proportion‚0.277587890625 9&Purity/1FRt1SRqqt0FBt/event_proportion‚0.143310546875 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9107142686843872 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9988771080970764 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7199500799179077 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4483119547367096 9"Purity/2FRt*SRqqt2FBt/event_purity‚0.8214285969734192 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9984472393989563 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5870780348777771 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17447948455810547 >&Purity/2FRt0SRqqt*FBt/event_proportion‚0.02082170732319355 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2968718111515045 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34917065501213074 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3697308897972107 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6324979066848755 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01607840321958065 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6189560294151306 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658390998840332 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8668831586837769 ?&Purity/1FRt0SRqqt1FBt/event_proportion‚0.009905134327709675 & Purity/1FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011335100047290325 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5692307949066162 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt0SRqqt2FBt/event_proportion‚0.011125837452709675 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38531824946403503 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6772319674491882 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991394281387329 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006905692163854837 *jet/accuracy_2_of_2‚0.1727919578552246 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4678083062171936 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9035559892654419 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988095164299011 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5199497938156128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3607352077960968 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.36255887150764465 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019000934436917305 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01949637196958065 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3582589030265808 7"Purity/0FRt*SRqqt1FBt/event_purity‚0.87553870677948 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053127773106098175 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17858301103115082 8&Purity/1FRt0SRqqt*FBt/event_proportion‚ 0.46142578125 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7172498106956482 ,&Purity/2FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7912930250167847 /particle/accuracy_3_of_3‚0.9993678331375122 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9986957311630249 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2874815762042999 A&Purity/1FRt1SRqqt2FBt/event_proportion‚0.00013950893480796367 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02336774580180645 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7769578695297241 8&Purity/0FRt*SRqqt2FBt/event_proportion‚ 0.00830078125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0301688052713871 (particle/accuracy‚0.8548176884651184 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9955520629882812 & Purity/1FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685064911842346 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7956777811050415 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.881477952003479 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6383249163627625 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8373957276344299 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5180065035820007 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7072649002075195 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4583333134651184 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27085497975349426 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6292827725410461 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1492745578289032 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6548104882240295 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3882982134819031 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305374979972839 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.09070056676864624 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.0414341501891613 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0017438615905120969 ?&Purity/2FRt1SRqqt*FBt/event_proportion‚0.013706752099096775 particle/accuracy_2_of_3‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026956509798765182 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8636363744735718 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0745496079325676 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2793044149875641 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31152063608169556 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029196735471487045 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6436420679092407 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6915340423583984 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4590897262096405 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0033482143189758062 6&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.052734375 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 particle/accuracy_1_of_3‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.36897221207618713 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7717949151992798 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.76884925365448 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46062731742858887 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0008021763642318547 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9530901312828064 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7517768740653992 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6146414875984192 +jet/accuracy_1_of_1‚0.37259361147880554 trainer/global_step‚13999 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9982143044471741 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6436420679092407 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00038364954525604844 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7073219418525696 +jet/accuracy_3_of_3‚0.07885482162237167 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3557097017765045 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2166224867105484 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.012904576025903225 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7168222069740295 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4015151858329773 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9487304091453552 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38253480195999146 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.0535365529358387 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6316066384315491 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8034074902534485 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.881477952003479 *jet/accuracy_2_of_3‚0.6057040095329285 _runtime‚25435.07346142 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9971320033073425 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0419573113322258 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47727271914482117 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7822175025939941 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6399146914482117 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6106577515602112 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0228097103536129 ,jet/accuracy_4_of_4‚0.018756305798888206 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8092294931411743 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30919167399406433 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2291138917207718 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.35776278376579285 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9640940427780151 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6884050369262695 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45512819290161133 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007533482275903225 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16212734580039978 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7963857650756836 *particle/sensitivity‚0.641884446144104 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6648514866828918 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9107142686843872 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0597795769572258 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00034877232974395156 & Purity/1FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1503138542175293 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19065406918525696 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9964285492897034 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02326311357319355 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3715417981147766 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8214285969734192 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00038364954525604844 _step‚1169 =&Purity/*FRt*SRqqt0FBt/event_proportion‚0.8812779188156128 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9987684488296509 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0179617740213871 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7358757853507996 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9791666865348816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0469098761677742 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3903312683105469 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008021763642318547 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019000934436917305 *jet/accuracy_1_of_3‚0.9387781023979187 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0416666679084301 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6007893681526184 epoch‚49 % _timestamp‚1.7848646717154095e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0009068080107681453 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9979591965675354 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7218963503837585 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.3000674247741699 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5463929772377014 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0314369834959507 >&Purity/1FRt2SRqqt0FBt/event_proportion‚0.02301897294819355 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2700788676738739 particle/accuracy_3_of_4‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8636363744735718‘ [ÔZRž: uub1bs2mj@ Ÿ¿‹ÓÀ÷ä0`Trainer.fit` stopped: `max_epochs=50` reached. ·—ß¡Ÿ: uub1bs2mjŽ  ¿‹Ó¸Åí“|Epoch 49/49 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 280/280 0:08:05 • 0:00:00 0.58it/s v_num: bs2m}0ÝP% : uub1bs2mj  ¿‹Óèó‚” O34¡:/ _wandbruntime‚25436  _runtime‚25436ŸÙ§Û¢:‚(’ÜÆ